Deploy vCluster on GKE

This guide provides step-by-step instructions for deploying vCluster on Google Kubernetes Engine (GKE).

Prerequisites

Before starting, ensure you have the following tools installed:

• kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.
• vCluster CLI
  Homebrew

  brew install loft-sh/tap/vcluster

  The binaries in the tap are signed using the Sigstore framework for enhanced security.

  Mac (Intel/AMD)

  curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

  Mac (Silicon/ARM)

  curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

  Linux (AMD)

  curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

  Linux (ARM)

  curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

  Download Binary

  Download the binary for your platform from the GitHub Releases page and add this binary to your $PATH.

  Windows Powershell

  md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';
  Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe;
  $env:Path += ";" + $Env:APPDATA + "\vcluster";
  [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);

  Reboot Required

  You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below).

  Check Environment Variable $PATH

  Line 4 of this install script adds the install directory %APPDATA%\vcluster to the $PATHenvironment variable. This is only effective for the current Powershell session, i.e. when opening a new terminal window,vcluster may not be found.

  Make sure to add the folder %APPDATA%\vcluster to the PATH environment variable after installing vcluster CLI via Powershell. Afterward, a reboot might be necessary.

  Confirm that you've installed the correct version of the vCluster CLI.

  vcluster --version

• Google Cloud SDK (gcloud CLI)
  note

  Ensure you have the necessary IAM permissions to create clusters and manage cloud services.

Create GKE cluster

Start by creating a zonal GKE cluster using the gcloud CLI. First, set up your environment variables:

tip

Project ID can be found in the Google Cloud Console under the project name. Alternatively use gcloud project list to list all projects and their IDs. To check which project is active, use gcloud config get-value project.

Modify the following with your specific values to generate a copyable command:

PROJECT_ID

CLUSTER_NAME

ZONE

MACHINE_TYPE

export PROJECT_ID=development
export CLUSTER_NAME=vcluster-demo
export ZONE=europe-west1-b
export MACHINE_TYPE=e2-standard-4

Configure gcloud and enable the required APIs and set default project:

Configure gcloud

gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com

Create the cluster:

Create GKE cluster

gcloud container clusters create $CLUSTER_NAME \
    --zone $ZONE \
    --machine-type $MACHINE_TYPE \
    --num-nodes 2

info

This process typically takes about 10-15 minutes.

This command creates a GKE cluster named vcluster-demo in the europe-west1-b zone with two nodes of type e2-standard-4.

kubeconfig update

This command automatically updates your kubeconfig file with the new cluster configuration.

Verify the host cluster creation

Verify the cluster by listing the nodes:

List cluster nodes

kubectl get nodes

You should see output similar to:

NAME           LOCATION        MASTER_VERSION      MASTER_IP      MACHINE_TYPE   NODE_VERSION        NUM_NODES  STATUS
vcluster-demo  europe-west1-b  1.30.5-gke.1443001  35.187.66.218  e2-standard-4  1.30.5-gke.1443001  2          RUNNING

Predeployment configuration options

Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options.

Control Plane Options

Decide the various options of how you want your control plane deployed:

• High availability - Run multiple copies of vCluster components.
• Rootless mode - Deploy the vCluster pod without root access to the host cluster.
• Backing Store - Decide how the data of your cluster is stored.
  Backing store options

  vCluster supports etcd or a relational database (using KINE) as the backend.This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case.

  vCluster supports the following datastore options:

  • Embedded SQLite (default with PersistentVolume (PV))
  • PostgreSQL
  • MySQL
  • MariaDB
  • etcd

  warning

  After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying.

  Backing store options

  Embedded SQLite (Default)

  This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use:

  controlPlane:
    backingStore:
      database:
        embedded:
          enabled: true

  Embedded SQLite (No PV)

  By default, vCluster stores its data in a PersistentVolumeClaim (PVC). Alternatively, you can use an emptyDir volume to store virtual cluster data.

  To use an emptyDir to store the data instead of a PersistentVolume, create a values.yaml with the following contents:

  controlPlane:
    statefulSet:
      persistence:
        volumeClaim:
          enabled: true

  Then upgrade or recreate the vCluster with:

  vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml

  Potential data loss

  This method should only be used for testing purposes, as data is lost upon pod recreation.

  Embedded etcd

  This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite:

  controlPlane:
    backingStore:
      etcd:
        embedded:
          enabled: true

  Deployed etcd

  This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store:

  controlPlane:
    backingStore:
      etcd:
        deploy:
          enabled: true

  MySQL / MariaDB

  The option for MySQL and MariaDB typically has the following format:

  controlPlane:
    backingStore:
      database:
        external:
          enabled: true
          dataSource: mysql://username:password@tcp(hostname:3306)/database-name

  If you specify a database name and it does not exist, the server attempts to create it.

  PostgreSQL

  The option for PostgreSQL typically has the following format:

  controlPlane:
    backingStore:
      database:
        external:
          enabled: true
          dataSource: postgres://username:password@hostname:port/database-name

  More advanced configuration parameters are available. For more information, see https://godoc.org/github.com/lib/pq.

  If you specify a database name and it does not exist, the server attempts to create it.

Worker Nodes

Decide where you want your worker nodes to come from:

• Nodes from the host cluster - (Default) All worker nodes of the shared host cluster are used by the virtual cluster and all resources are synced to the single namespace that the vCluster is deployed on.
  • Syncing Namespaces - Resources are synced to mapped namespaces on the host cluster.
  • Isolated workloads - Different options to isolate a workload in a vCluster.
• Private Nodes - Enable adding individual nodes to the virtual cluster.

Deploy vCluster on GKE

YAML configuration

If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated vcluster.yaml. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade.

All of the deployment options below have the following assumptions:

• A vcluster.yaml is provided. Refer to the vcluster.yaml reference docs to explore all configuration options. This file is optional and can be removed from the examples.
• The vCluster is called my-vcluster.
• The vCluster is be deployed into the team-x namespace.

vCluster CLI

The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters.

1. Install the vCluster CLI:

   Homebrew

   brew install loft-sh/tap/vcluster

   The binaries in the tap are signed using the Sigstore framework for enhanced security.

   Mac (Intel/AMD)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Mac (Silicon/ARM)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Linux (AMD)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Linux (ARM)

   curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

   Download Binary

   Download the binary for your platform from the GitHub Releases page and add this binary to your $PATH.

   Windows Powershell

   md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';
   Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe;
   $env:Path += ";" + $Env:APPDATA + "\vcluster";
   [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);

   Reboot Required

   You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below).

   Check Environment Variable $PATH

   Line 4 of this install script adds the install directory %APPDATA%\vcluster to the $PATHenvironment variable. This is only effective for the current Powershell session, i.e. when opening a new terminal window,vcluster may not be found.

   Make sure to add the folder %APPDATA%\vcluster to the PATH environment variable after installing vcluster CLI via Powershell. Afterward, a reboot might be necessary.

   Confirm that you've installed the correct version of the vCluster CLI.

   vcluster --version

2. Deploy vCluster:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   NAMESPACE

   CONFIG FILE

   vcluster create my-vcluster --namespace team-x --values vcluster.yaml

   note

   After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run kubectl commands against the virtual cluster.

Helm

Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows.

1. Deploy vCluster using the helm upgrade command:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   CONFIG FILE

   NAMESPACE

   helm upgrade --install my-vcluster vcluster \
     --values vcluster.yaml \
     --repo https://charts.loft.sh \
     --namespace team-x \
     --repository-config='' \
     --create-namespace

Terraform

You can use Terraform to deploy vCluster as code with version control and state management.

1. Create a main.tf file to define your vCluster deployment using the Terraform Helm provider:

   provider "helm" {
     kubernetes {
       config_path = "~/.kube/config"
     }
   }
   
   resource "helm_release" "my_vcluster" {
     name             = "my-vcluster"
     namespace        = "team-x"
     create_namespace = true
   
     repository       = "https://charts.loft.sh"
     chart            = "vcluster"
   
     # If you didn't create a vcluster.yaml, remove the values section.
     values = [
       file("${path.module}/vcluster.yaml")
     ]
   }

2. Install the required Helm provider and initialize Terraform:

   terraform init

3. Generate a plan to preview the changes:

   terraform plan

   
   

   Review the plan output to verify connectivity and proposed changes.

4. Deploy vCluster:

   terraform apply

Argo CD

ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories.

To deploy vCluster using ArgoCD, you need the following files:

• vcluster.yaml for your vCluster configuration options.
• <CLUSTER_NAME>-app.yaml for your ArgoCD Application definition. Replace <CLUSTER_NAME> with your actual cluster name.

1. Create the ArgoCD Application file <CLUSTER_NAME>-app.yaml, which references the vCluster Helm chart:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   NAMESPACE

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
   name: my-vcluster
   namespace: argocd
   spec:
   project: default
   source:
     chart: vcluster
     repoURL: https://charts.loft.sh
     helm:
       releaseName: my-vcluster
       valueFiles:
         - vcluster.yaml
   destination:
     server: https://kubernetes.default.svc
     namespace: team-x

2. Commit and push these files to your configured ArgoCD repository.

3. Sync your ArgoCD repository with your configured cluster:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   argocd app sync my-vcluster

Cluster API

Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the Cluster API Provider for vCluster documentation.

1. Install the clusterctl CLI.

2. Install the vCluster provider:

   clusterctl init --infrastructure vcluster:v0.2.0

3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster.

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   NAMESPACE

   CONFIG FILE

   export CLUSTER_NAME=my-vcluster
   export CLUSTER_NAMESPACE=team-x
   export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml)

4. Create the namespace for the vCluster using the exported variable:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAMESPACE

   kubectl create namespace team-x

5. Generate the required manifests and apply them using the exported variables:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAME

   CLUSTER NAMESPACE

   clusterctl generate cluster my-vcluster \
     --infrastructure vcluster \
     --target-namespace team-x \
   | kubectl apply -f -

   Kubernetes version

   The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the vcluster.yaml file based on your Kubernetes distribution.

6. Wait for vCluster to become ready by monitoring the vCluster custom resource status:

   Modify the following with your specific values to generate a copyable command:

   CLUSTER NAMESPACE

   CLUSTER NAME

   kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s

Next steps

Now that you have vCluster running on GKE, consider setting up the platform UI to mange your virtual clusters.

Workload Identity

Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

When using the platform you can easily enable Workload Identity.

Cleanup

If you deployed the GKE cluster with this tutorial, and want to clean up the resources, run the following command:

Clean up resources

gcloud container clusters delete $CLUSTER_NAME --zone $ZONE --quiet