Skip to main content

Isolation & Security

vcluster can drastically increase security in multi-tenancy clusters. vcluster provides you the three security benefits out of the box:

  • Full control-plane isolation with separate api endpoint and data storage
  • DNS isolation as vcluster workloads are not able to resolve any services of the host cluster
  • Guarantee that all workloads, services and other namespaced objects are created in a single namespace in the host cluster. If deployed with default settings, vcluster also ensures that no access to any cluster scoped object is required.

Besides these benefits, vcluster by default will not provide any workload or network isolation. Starting with version v0.7.0, vcluster has a feature called isolated mode, which you can enable to prevent vcluster workloads from breaking out of their virtual environment.

In general, we recommend to deploy a single vcluster into a namespace and then isolate the namespace, which is far easier than isolating multiple vclusters from each other in a single namespace.

Isolated Mode#

vcluster offers a feature called isolated mode to automatically isolate workloads in a virtual cluster. Isolated mode can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true:

# Creates a new vcluster with isolated workloads
vcluster create my-vcluster --isolate

This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

  1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as a privileged container or mount a host path will not be synced to the host cluster. Current valid options are either baseline (default in isolated mode) or restricted. This works for every Kubernetes version regardless of Pod Security Standard support, as this is implemented in vcluster directly. Rejected pods will stay pending in the vcluster and in newer Kubernetes version they will be denied by the admission controller as well.
  2. vcluster deploys a resource quota as well as a limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled, sane defaults for those 2 resources are chosen.
  3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads as well as the vcluster control plane to other pods in the host cluster. (only works if your host cluster CNI supports network policies)

You can adjust isolation settings through helm values. The default values are (also check values.yaml):

isolation:
enabled: false
podSecurityStandard: baseline
resourceQuota:
enabled: true
quota:
requests.cpu: 10
requests.memory: 20Gi
requests.storage: "100Gi"
requests.ephemeral-storage: 60Gi
limits.cpu: 20
limits.memory: 40Gi
limits.ephemeral-storage: 160Gi
services.nodeports: 0
services.loadbalancers: 1
count/endpoints: 40
count/pods: 20
count/services: 20
count/secrets: 100
count/configmaps: 100
count/persistentvolumeclaims: 20
scopeSelector:
matchExpressions:
scopes:
limitRange:
enabled: true
default:
ephemeral-storage: 8Gi
memory: 512Mi
cpu: "1"
defaultRequest:
ephemeral-storage: 3Gi
memory: 128Mi
cpu: 100m
networkPolicy:
enabled: true
outgoingConnections:
ipBlock:
cidr: 0.0.0.0/0
except:
- 100.64.0.0/10
- 127.0.0.0/8
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16

Workload Isolation#

vcluster by default will not isolate any workloads in the host cluster and only ensures that those are deployed in the same namespace. However, isolating workloads in a single namespace can be done with in-built Kubernetes features or using the isolated mode shown above.

Resource Quota & Limit Range#

To ensure a vcluster will not consume too many resources in the host cluster, you can use a single ResourceQuota in the namespace where the virtual cluster is running. This could look like:

apiVersion: v1
kind: ResourceQuota
metadata:
name: vcluster-quota
spec:
hard:
cpu: "10"
memory: 20Gi
pods: "10"

This allows the vcluster and all of the pods deployed inside it to only consume up to 10 vCores, 20GB of memory or to have 10 pods at maximum. If you use a resource quota, you probably also want to use a LimitRange that makes sure that needed resources are defined for each pod. For example:

apiVersion: v1
kind: LimitRange
metadata:
name: vcluster-limit-range
spec:
limits:
- default:
memory: 512Mi
cpu: "1"
defaultRequest:
memory: 128Mi
cpu: 100m
type: Container

This limit range would ensure that containers that do not set resources.requests and resources.limits would get appropriate limits set automatically.

Pod Security#

Besides restricting pod resources, it's also necessary to disallow certain potential harmful pod configurations, such as privileged pods or pods that use hostPath. If you are using Kubernetes v1.23 or higher, you can restrict the namespace where the virtual cluster is running in via the Pod Security Admission Controller:

apiVersion: v1
kind: Namespace
metadata:
name: my-vcluster-namespace
labels:
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted

To see all supported levels and modes, please take a look at the Kubernetes docs.

If you are using below Kubernetes v1.23 clusters, you can use the deprecated PodSecurityPolicies to disallow critical workloads.

If you want more control over this, you can also use an admission controller, that let's you define your own policies, such as OPA, jsPolicy or Kyverno.

Advanced Isolation#

Besides this basic workload isolation, you could also dive into more advanced isolation methods, such as isolating the workloads on separate nodes or through another container runtime. Using different nodes for your vcluster workloads can be accomplished through the --node-selector flag on vcluster syncer.

You should also be aware that pods created in the vcluster will set their tolerations, which will affect scheduling decisions. To prevent the pods from being scheduled to the undesirable nodes you can use the --node-selector flag or admission controller as mentioned above.

Network Isolation#

Workloads created by vcluster will be able to communicate with other workloads in the host cluster through their cluster ips. This can be sometimes beneficial if you want to purposely access a host cluster service, which is a good method to share services between vclusters. However, you often want to isolate namespaces and do not want the pods running inside vcluster to have access to other workloads in the host cluster. This requirement can be accomplished by using Network Policies for the namespace where vcluster is installed in or using the isolated mode shown above.

info

Network policies do not work in all Kubernetes clusters and need to be supported by the underlying CNI plugin.

Other Topics#

Running as non root#

vcluster is able to be ran as a non root user. Steps below show how to set the desired UID for syncer and control plane. The syncer also passes this UID down to the vcluster DNS deployment.

Create a values.yaml file with the following lines:

#fsGroup: 12345
securityContext:
runAsUser: 12345
runAsGroup: 12345
runAsNonRoot: true

Then create the vcluster with the following command:

vcluster create my-vcluster -f values.yaml

Workload & Network Isolation within the vcluster#

The above mentioned methods also work for isolating workloads inside the vcluster itself, as you can just deploy resource quotas, limit ranges, admission controllers and network policies in there. To allow network policies to function correctly, you'll need to enable this in vcluster itself though.

Edit this page