Other Kubernetes distributions

By default, vcluster will use k3s as virtual Kubernetes cluster, which is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.

However, vcluster is not tied to a specific distribution and should work with all certified Kubernetes distributions. By default, we recommend to use k3s, because it has a small footprint and widely adopted, but if your use case requires a different k8s distribution, vcluster currently also supports k0s or vanilla k8s. If that is also not enough, you can also add your custom Kubernetes distribution as outlined below.

k0s#

k0s is an all-inclusive Kubernetes distribution, which is configured with all of the features needed to build a Kubernetes cluster and packaged as a single binary for ease of use. vcluster supports k0s as backing virtual Kubernetes cluster.

In order to use k0s as backing cluster, create a vcluster with the following command:

vcluster create my-vcluster --distro k0s

Start using it:

kubectl get ns
...

Behind the scenes a different helm chart will be deployed (vcluster-k0s), that holds specific configuration to support k0s. Check the github repository for all available chart options.

Vanilla k8s#

When choosing this option, vcluster will deploy a separate etcd cluster, kubernetes controller manager and api server alongside the vcluster hypervisor.

In order to use vanilla k8s as backing cluster, create a vcluster with the following command:

vcluster create my-vcluster --distro k8s

Connect to the vcluster and start using it:

kubectl get ns
...

Behind the scenes a different helm chart will be deployed (vcluster-k8s), that holds specific configuration to support vanilla k8s. Check the github repository for all available chart options.

High Available Vanilla k8s#

Enabling High Availability#

In order to run vcluster in high availability mode, create a values.yaml in the following form:

# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3

# Scale up etcd
etcd:
  replicas: 3

# Scale up controller manager
controller:
  replicas: 3

# Scale up api server
api:
  replicas: 3

# Scale up DNS server
coredns:
  replicas: 3

Then create the vcluster with the following command:

vcluster create ... --connect=false --distro k8s -f values.yaml

Check that vcluster including the control plane are running correctly:

kubectl get po -n vcluster
NAME                                                READY   STATUS    RESTARTS   AGE
coredns-6ff7df994d-m5pcd-x-kube-system-x-vcluster   1/1     Running   0          23m
coredns-6ff7df994d-dfgjb-x-kube-system-x-vcluster   1/1     Running   0          23m
coredns-6ff7df994d-weuir-x-kube-system-x-vcluster   1/1     Running   0          23m
vcluster-9d88f577-m55xf                             1/1     Running   0          30m
vcluster-9d88f577-drsxz                             1/1     Running   0          30m
vcluster-9d88f577-maslo                             1/1     Running   0          30m
vcluster-api-66bfc4cf94-cp28t                       1/1     Running   0          30m
vcluster-api-66bfc4cf94-drnll                       1/1     Running   0          30m
vcluster-api-66bfc4cf94-jfbnn                       1/1     Running   0          30m
vcluster-controller-b4cd55bb6-9mvc4                 1/1     Running   0          30m
vcluster-controller-b4cd55bb6-bmfdj                 1/1     Running   0          30m
vcluster-controller-b4cd55bb6-kcxr7                 1/1     Running   0          30m
vcluster-etcd-0                                     1/1     Running   0          30m
vcluster-etcd-1                                     1/1     Running   0          29m
vcluster-etcd-2                                     1/1     Running   0          29m

Now connect to the vcluster:

vcluster connect vcluster-1 -n host-namespace-1

# Then execute in a new terminal
export KUBECONFIG=kubeconfig.yaml
kubectl get ns
...

Enable HA in rootless mode#

Rootless mode means running vcluster without root user privileges in container, making host k8s cluster more secure. You can find more about rootless mode here.

Below is HA configuration for running rootless vcluster with vanilla Kubernetes distribution.

# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: true
    runAsGroup: 12345
    runAsNonRoot: true
    runAsUser: 12345
    seccompProfile:
      type: RuntimeDefault

# Scale up etcd
etcd:
  replicas: 3
  fsGroup: 12345
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: true
    runAsGroup: 12345
    runAsNonRoot: true
    runAsUser: 12345
    seccompProfile:
      type: RuntimeDefault

# Scale up controller manager
controller:
  replicas: 3
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: true
    runAsGroup: 12345
    runAsNonRoot: true
    runAsUser: 12345
    seccompProfile:
      type: RuntimeDefault

# Scale up api server
api:
  replicas: 3
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: true
    runAsGroup: 12345
    runAsNonRoot: true
    runAsUser: 12345
    seccompProfile:
      type: RuntimeDefault

# Scale up DNS server
coredns:
  replicas: 3
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: true
    runAsGroup: 12345
    runAsNonRoot: true
    runAsUser: 12345
    seccompProfile:
      type: RuntimeDefault

Check the github repository for all available chart options.

Other Distributions#

vcluster has no dependencies on any specific Kubernetes distribution, so you should be able to run it with most certified Kubernetes distributions. One requirement vcluster has, is that the distribution can be deployed without a scheduler and kubelet, meaning that vcluster just requires the api server, controller manager and data storage of the distribution.

For single binary distributions, such as k3s or k0s, extra bundled components can usually be disabled through flags, for multi binary distributions, such as vanilla k8s, you just need to deploy the virtual control plane with api server, controller manager and usually etcd. Most multi binary distributions work by just overriding the images of the k8s chart in a values.yaml, e.g.:

api:
  image: custom-domain.com/custom-kube-apiserver:v1.21.5
controller:
  image: custom-domain.com/custom-kube-controller-manager:v1.21.5
etcd:
  image: custom-domain.com/custom-etcd:v3.4.16

And then deploy vcluster with:

vcluster create my-vcluster -n test --distro k8s -f values.yaml

If you want to create a separate chart for the Kubernetes distribution, a good starting point is to copy one of our distro charts and then modifying it to work with your distribution. vcluster only needs the following information from the virtual Kubernetes distribution to function properly:

The api server central authority certificate (usually found at /pki/ca.crt)
The api server central authority key (usually found at /pki/ca.key)
An admin kube config to contact the virtual Kubernetes control plane (usually found at /pki/admin.conf)

For multi binary distributions, vcluster can even create those with a pre-install hook as found in the k8s chart.

In general, if you need vcluster to support another Kubernetes distribution, we are always happy to help you or accept a pull request in our github repository.