Exposing vcluster (ingress etc.)

By default, vcluster is only reachable via port-forwarding in remote clusters. However, this means that you need access to the host cluster, where the vcluster is running, in order to access it. To directly access vcluster without port-forwarding, you can use one of the following methods:

LoadBalancer service
NodePort service
Ingress
In-Cluster

Local Kubernetes Clusters

If you are using a local Kubernetes cluster, such as docker-desktop, rancher-desktop, KinD or minikube, vcluster will automatically connect to it without the need of port-forwarding.

LoadBalancer service#

The easiest way is to use the flag --expose in vcluster create to tell vcluster to use a LoadBalancer service:

# Create a new vcluster with a LoadBalancer
vcluster create my-vcluster --expose

# Run any kube command in the vcluster
kubectl get ns

Thats it, your vcluster is now externally reachable through a LoadBalancer service.

Check the costs first

Even though using a LoadBalancer is the easiest option, if you use a cloud provider it will be costly to create one Loadbalancer per cluster. Check your cloud vendor about the cost of each LoadBalancer. In general using an Ingress is the most cost effective method.

Manual LoadBalancer service creation#

Instead of using the built-in flag --expose, you can also create the following load-balancer.yaml for a vcluster called my-vcluster in the namespace my-vcluster yourself:

apiVersion: v1
kind: Service
metadata:
  name: vcluster-loadbalancer
  namespace: my-vcluster
spec:
  selector:
    app: vcluster
    release: my-vcluster
  ports:
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP
  type: LoadBalancer

Create the resource in the namespace via:

kubectl apply -f load-balancer.yaml

Find out the external ip via kubectl get svc vcluster-loadbalancer -n my-vcluster:

kubectl get svc vcluster-loadbalancer -n my-vcluster
NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)         AGE
vcluster-loadbalancer   LoadBalancer   10.68.9.239   x.x.x.x        443:32678/TCP   7m15s

Now create a values.yaml to create the vcluster with:

syncer:
  extraArgs:
  - --tls-san=x.x.x.x

Create the virtual cluster with:

vcluster create my-vcluster -n my-vcluster --connect=false -f values.yaml

Update the current kube config via:

# Update the current kube config (or use --update-current=false to create a separate one)
vcluster connect my-vcluster -n my-vcluster --server=https://x.x.x.x

Access the vcluster:

# Run any kube context command
kubectl get ns

NodePort service#

You can also expose the vcluster via a NodePort service. Create the following nodeport.yaml for a vcluster called my-vcluster in the namespace my-vcluster:

apiVersion: v1
kind: Service
metadata:
  name: vcluster-nodeport
  namespace: my-vcluster
spec:
  selector:
    app: vcluster
    release: my-vcluster
  ports:
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP
  type: NodePort

Create the resource in the namespace via:

kubectl apply -f nodeport.yaml

Find out the external port via kubectl get svc vcluster-nodeport -n my-vcluster:

kubectl get svc vcluster-nodeport -n my-vcluster
NAME                TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
vcluster-nodeport   NodePort   10.68.9.75   <none>        443:31992/TCP   85s

Find out the node ips via kubectl get nodes -o wide:

NAME                                       STATUS   ROLES    AGE   VERSION            INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-cluster-1-default-pool-8f0bb8bb-p6wx   Ready    <none>   6d    v1.20.6-gke.1000   10.156.0.14   x.x.x.x         Container-Optimized OS from Google   5.4.104+         containerd://1.4.3
gke-cluster-1-default-pool-8f0bb8bb-vl79   Ready    <none>   6d    v1.20.6-gke.1000   10.156.0.15   y.y.y.y         Container-Optimized OS from Google   5.4.104+         containerd://1.4.3
gke-cluster-1-default-pool-8f0bb8bb-wpkp   Ready    <none>   6d    v1.20.6-gke.1000   10.156.0.16   z.z.z.z         Container-Optimized OS from Google   5.4.104+         containerd://1.4.3

Now create a values.yaml to create the vcluster with:

syncer:
  extraArgs:
  - --tls-san=x.x.x.x,y.y.y.y,z.z.z.z

Create the virtual cluster with:

vcluster create my-vcluster -n my-vcluster --connect=false -f values.yaml

Retrieve the kube config via:

vcluster connect my-vcluster -n my-vcluster --update-current=false --server=https://x.x.x.x

Access the vcluster:

export KUBECONFIG=./kubeconfig.yaml

# Run any kube context command
kubectl get ns

Ingress#

An Ingress Controller with SSL passthrough support will provide the best user experience, but there is a workaround if this feature is not natively supported.

Make sure your ingress controller is installed and healthy on the cluster that will host your virtual clusters. Create the following ingress.yaml for a vcluster called my-vcluster in the namespace my-vcluster:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # We need the ingress to pass through ssl traffic to the vcluster
    # This only works for the nginx-ingress (enable via --enable-ssl-passthrough
    # https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough )
    # for other ingress controllers please check their respective documentation.
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: vcluster-ingress
  namespace: my-vcluster
spec:
  ingressClassName: nginx # use your ingress class name
  rules:
  - host: my-vcluster.example.com
    http:
      paths:
      - backend:
          service:
            name: my-vcluster
            port: 
              number: 443
        path: /
        pathType: ImplementationSpecific

Create the resource in the namespace via:

kubectl apply -f ingress.yaml

Enable SSL Passthrough Feature

If you are using the ingress nginx controller, please make sure you have enabled the SSL passthrough feature as it is disabled by default.

SSL Passthrough required

In order for this ingress to work correctly, you will need to enable SSL passthrough as TLS termination has to happen at the vcluster level and not ingress controller level. If you cannot do that, please take a look below for using an ingress without ssl passthrough.

Now create a values.yaml to create the vcluster with:

syncer:
  extraArgs:
  - --tls-san=my-vcluster.example.com

Create the virtual cluster with:

vcluster create my-vcluster -n my-vcluster --connect=false -f values.yaml

Retrieve the kube config via:

vcluster connect my-vcluster -n my-vcluster --update-current=false --server=https://my-vcluster.example.com

Access the vcluster:

export KUBECONFIG=./kubeconfig.yaml

# Run any kubectl command
kubectl get ns

Ingress without SSL-Passthrough#

If you cannot configure your ingress controller to use ssl-passthrough, you can also create an ingress similar to this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: vcluster-ingress
  namespace: my-vcluster
spec:
  ingressClassName: nginx # use your ingress class name
  rules:
  - host: my-vcluster.example.com
    http:
      paths:
      - backend:
          service:
            name: my-vcluster
            port: 
              number: 443
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts: my-vcluster.example.com

With this configuration you will need to use service account authentication in order to connect as the ingress controller won't be able to resolve the client-cert and client-key which is used by default as authentication method. To create a kube config that uses a service account, please run the following command:

vcluster connect my-vcluster -n my-vcluster --server=https://my-vcluster.example.com --service-account admin --cluster-role cluster-admin --insecure

Then access the vcluster:

# Run any kubectl command
kubectl get ns

In-Cluster#

In order to access the virtual cluster from within the host cluster, you can directly connect to the vcluster service. Make sure you can access that service and then create a kube config in the following form:

vcluster connect my-vcluster -n my-vcluster --server=my-vcluster.my-vcluster --insecure --update-current=false

Now access the virtual cluster with:

export KUBECONFIG=./kubeconfig.yaml

# Run any kubectl command
kubectl get ns