Deploy in high availability

By default, vCluster runs one instance of each of its components. This deployment method is recommended for any uses cases that are very ephemeral (e.g. dev environments, CI/CD, etc.), but for production use cases, it's recommended to run vCluster with more redundancy. We recommend deploying vCluster in high availability (HA) in order to run multiple copies of the vCluster components so that the virtual cluster is more resistant to failures.

Enabling high availability

When running vCluster with high availability, the default backing store option (i.e. embedded database) must be changed to any of the other backing store options.

Deployed etcd YAML example

Running in HA with deployed etcd

controlPlane:
  # Deploy etcd with 3 replicas
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
  # Deploy vCluster with 3 replicas
  statefulSet:
    highAvailability:
      replicas: 3

Embedded etcd YAML example

Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Running in HA with embedded etcd

controlPlane:
  # Deploy etcd with 3 replicas
  backingStore:
    etcd:
      embedded:
        enabled: true
  # Deploy vCluster with 3 replicas
  statefulSet:
    highAvailability:
      replicas: 3

Other predeployment configuration options

Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options.

Control Plane Options

Decide the various options of how you want your control plane deployed:

• High availability - Run multiple copies of vCluster components.
• Rootless mode - Deploy the vCluster pod without root access to the host cluster.
• Backing Store - Decide how the data of your cluster is stored.
  Backing store options

  vCluster supports etcd or a relational database (using KINE) as the backend.This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case.

  vCluster supports the following datastore options:

  • Embedded SQLite (default with PersistentVolume (PV))
  • PostgreSQL
  • MySQL
  • MariaDB
  • etcd

  warning

  After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying.

  Backing store options

  Embedded SQLite (Default)

  This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use:

  controlPlane:
    backingStore:
      database:
        embedded:
          enabled: true

  Embedded SQLite (No PV)

  By default, vCluster stores its data in a PersistentVolumeClaim (PVC). Alternatively, you can use an emptyDir volume to store virtual cluster data.

  To use an emptyDir to store the data instead of a PersistentVolume, create a values.yaml with the following contents:

  controlPlane:
    statefulSet:
      persistence:
        volumeClaim:
          enabled: true

  Then upgrade or recreate the vCluster with:

  vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml

  Potential data loss

  This method should only be used for testing purposes, as data is lost upon pod recreation.

  Embedded etcd

  This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite:

  controlPlane:
    backingStore:
      etcd:
        embedded:
          enabled: true

  Deployed etcd

  This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store:

  controlPlane:
    backingStore:
      etcd:
        deploy:
          enabled: true

  MySQL / MariaDB

  The option for MySQL and MariaDB typically has the following format:

  controlPlane:
    backingStore:
      database:
        external:
          enabled: true
          dataSource: mysql://username:password@tcp(hostname:3306)/database-name

  If you specify a database name and it does not exist, the server attempts to create it.

  PostgreSQL

  The option for PostgreSQL typically has the following format:

  controlPlane:
    backingStore:
      database:
        external:
          enabled: true
          dataSource: postgres://username:password@hostname:port/database-name

  More advanced configuration parameters are available. For more information, see https://godoc.org/github.com/lib/pq.

  If you specify a database name and it does not exist, the server attempts to create it.

Worker Nodes

Decide where you want your worker nodes to come from:

• Nodes from the host cluster - (Default) All worker nodes of the shared host cluster are used by the virtual cluster and all resources are synced to the single namespace that the vCluster is deployed on.
  • Syncing Namespaces - Resources are synced to mapped namespaces on the host cluster.
  • Isolated workloads - Different options to isolate a workload in a vCluster.
• Private Nodes - Enable adding individual nodes to the virtual cluster.

Deploy vCluster with HA

YAML configuration

If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated vcluster.yaml. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade.

All of the deployment options below have the following assumptions:

• A vcluster.yaml is provided. Refer to the vcluster.yaml reference docs to explore all configuration options. This file is optional and can be removed from the examples.
• The vCluster is called my-vcluster.
• The vCluster is be deployed into the team-x namespace.

vCluster CLI

The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters.

1. Install the vCluster CLI:

   Homebrew

   brew install loft-sh/tap/vcluster

   The binaries in the tap are signed using the Sigstore framework for enhanced security.

   Mac (Intel/AMD)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Mac (Silicon/ARM)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Linux (AMD)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Linux (ARM)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Download Binary

   Download the binary for your platform from the GitHub Releases page and add this binary to your $PATH.

   Windows Powershell

   md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';
   Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe;
   $env:Path += ";" + $Env:APPDATA + "\vcluster";
   [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);

   Reboot Required

   You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below).

   Check Environment Variable $PATH

   Line 4 of this install script adds the install directory %APPDATA%\vcluster to the $PATHenvironment variable. This is only effective for the current Powershell session, i.e. when opening a new terminal window,vcluster may not be found.

   Make sure to add the folder %APPDATA%\vcluster to the PATH environment variable after installing vcluster CLI via Powershell. Afterward, a reboot might be necessary.

   Confirm that you've installed the correct version of the vCluster CLI.

   vcluster --version

2. Deploy vCluster:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   NAMESPACE

   CONFIG FILE

   vcluster create my-vcluster --namespace team-x --values vcluster.yaml

   note

   After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run kubectl commands against the virtual cluster.

Helm

Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows.

1. Deploy vCluster using the helm upgrade command:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   CONFIG FILE

   NAMESPACE

   helm upgrade --install my-vcluster vcluster \
     --values vcluster.yaml \
     --repo https://charts.loft.sh \
     --namespace team-x \
     --repository-config='' \
     --create-namespace

Terraform

You can use Terraform to deploy vCluster as code with version control and state management.

1. Create a main.tf file to define your vCluster deployment using the Terraform Helm provider:

   provider "helm" {
     kubernetes {
       config_path = "~/.kube/config"
     }
   }
   
   resource "helm_release" "my_vcluster" {
     name             = "my-vcluster"
     namespace        = "team-x"
     create_namespace = true
   
     repository       = "https://charts.loft.sh"
     chart            = "vcluster"
   
     # If you didn't create a vcluster.yaml, remove the values section.
     values = [
       file("${path.module}/vcluster.yaml")
     ]
   }

2. Install the required Helm provider and initialize Terraform:

   terraform init

3. Generate a plan to preview the changes:

   terraform plan

   
   

   Review the plan output to verify connectivity and proposed changes.

4. Deploy vCluster:

   terraform apply

Argo CD

ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories.

To deploy vCluster using ArgoCD, you need the following files:

• vcluster.yaml for your vCluster configuration options.
• <CLUSTER_NAME>-app.yaml for your ArgoCD Application definition. Replace <CLUSTER_NAME> with your actual cluster name.

1. Create the ArgoCD Application file <CLUSTER_NAME>-app.yaml, which references the vCluster Helm chart:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   NAMESPACE

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
   name: my-vcluster
   namespace: argocd
   spec:
   project: default
   source:
     chart: vcluster
     repoURL: https://charts.loft.sh
     helm:
       releaseName: my-vcluster
       valueFiles:
         - vcluster.yaml
   destination:
     server: https://kubernetes.default.svc
     namespace: team-x

2. Commit and push these files to your configured ArgoCD repository.

3. Sync your ArgoCD repository with your configured cluster:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   argocd app sync my-vcluster

Cluster API

Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the Cluster API Provider for vCluster documentation.

1. Install the clusterctl CLI.

2. Install the vCluster provider:

   clusterctl init --infrastructure vcluster:v0.2.0

3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster.

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   NAMESPACE

   CONFIG FILE

   export CLUSTER_NAME=my-vcluster
   export CLUSTER_NAMESPACE=team-x
   export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml)

4. Create the namespace for the vCluster using the exported variable:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAMESPACE

   kubectl create namespace team-x

5. Generate the required manifests and apply them using the exported variables:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   CLUSTER NAMESPACE

   clusterctl generate cluster my-vcluster \
     --infrastructure vcluster \
     --target-namespace team-x \
   | kubectl apply -f -

   Kubernetes version

   The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the vcluster.yaml file based on your Kubernetes distribution.

6. Wait for vCluster to become ready by monitoring the vCluster custom resource status:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAMESPACE

   CLUSTER NAME

   kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s

Check the pods of the vCluster on the host cluster

Ensure that your kubecontext is set to the host cluster.

warning

If you've used the vCluster CLI, your kubecontext automatically switches and connects to the virtual cluster.

You can get the pods in the namespace of the virtual cluster. The namespace is called vcluster-my-vcluster in the host cluster, and contains the components of the vCluster.

kubectl get pods -n vcluster-my-vcluster

Example using the deployed etcd yaml:

NAME                                      READY   STATUS    RESTARTS   AGE
my-vcluster-7c5c5844c5-27j2v              0/1     Running   0          20s
my-vcluster-7c5c5844c5-gb2sm              0/1     Running   0          20s
my-vcluster-7c5c5844c5-pwn7k              0/1     Running   0          20s
my-vcluster-etcd-0                        0/1     Running   0          20s
my-vcluster-etcd-1                        0/1     Running   0          20s
my-vcluster-etcd-2                        0/1     Running   0          20s

There are three replicas running of each component (control plane and etcd) of the vCluster. If one API server pod goes down, the vCluster continues working.

In order to see which nodes in the host cluster that these pods were scheduled on, add the -o wide flag to the kubectl get pods command:

kubectl get pods -n vcluster-my-vcluster -o wide

The hostnames of the nodes will be listed in the NODES column.