Why the nodes/proxy Kubernetes RCE Does Not Apply to vCluster

In a recent blog post, Graham Helton talks about the danger of using nodes/proxy RBAC permissions in Kubernetes. The vulnerability itself was not considered a CVE from Kubernetes itself and instead labelled as “working as intended”. Since vCluster occurs in the blog post as a helm chart being affected by this and customers of ours were curious on how this is used in vCluster, we decided to write a small blog post about it.

We have concluded that vCluster is not compromised in any way - and in fact, can provide more security in this context than vanilla Kubernetes, when using features that require the nodes/proxy permission.

What is dangerous about the nodes/proxy permission?

In Kubernetes permissions work additively, meaning that a user, group or serviceaccount starts with no permissions at all and then roles or cluster roles add rules with allowed actions to it. Each rule is defined by the following:

apiGroups: the api group of the resources the rule should allow, this can be for example apps or my-custom-api-group.company.com. Since nodes are part of the core api group, which is empty you define that as “”

resources: the resources the rule should apply to. In this case the resource would be `nodes`, but could also be any other Kubernetes resource such as pods etc. Subresources, such as logs, exec or proxy are defined by specifying the resource with the subresource like this: pods/exec, pods/log or nodes/proxy

verbs: the action the rule should apply to. Basically Kubernetes maps http methods to verbs, but these could in theory also be custom defined.

Besides defining what rules should be allowed inside a Kubernetes cluster for a given user, it’s also important to define that these rules can either apply on the namespace or cluster level. Depending on if they are defined in a ClusterRole or Role, they apply either globally or just locally in a namespace. Since nodes in Kubernetes are cluster scoped resources, you have to define the nodes/proxy permission in a ClusterRole and bind that via a ClusterRoleBinding to a serviceaccount, user or group.

Obviously, there are certain very privileged actions within a Kubernetes cluster you wouldn’t want to allow globally or even locally in a namespace. The blog post from Graham talks about nodes/proxy being dangerous since it's quite often used for monitoring stacks as it's a simple way to request kubelet information through the Kubernetes api server or on the kubelet itself. The confusing and potentially vulnerable part comes from the usage of the verb with nodes/proxy, where all listed applications, including vCluster, always only require get and not any of the writing verbs such as create,update or delete.

However, and that is the argument of Graham, if you are reaching out to the kubelet directly, there are ways to execute commands just with the get verb which shouldn’t be possible because it implies read-only behaviour

This is possible because the kubelet api does an authorization check for the websocket handshake which implies the get verb. The K8s api proxy doesn’t support websockets for nodes/proxy which is why the problem only exists when you contact the kubelet api directly.

Once the handshake is complete - using only get - you can execute arbitrary commands inside the connection like so:

wss://NODE_IP/exec/NAMESPACE/POD/CONTAINER?output=1&error=1&command=COMMAND

Why does vCluster need nodes/proxy?

By default, vCluster does not require that permission and only uses a namespace based Role to configure RBAC permissions excluding the nodes/proxy one.

However, vCluster has a feature called proxy kubelet endpoints that, when enabled, will add the nodes/proxy permission to vCluster itself. The feature allows deploying other applications that require this permission as well, e.g. prometheus stack or opentelemetry, to use it from within the vCluster to scrape node metrics. vCluster will filter and rewrite certain endpoints to make that possible.

Most importantly - vCluster does not allow the websocket protocol for the kubelet proxy and only accepts standard http requests. That in turn means that any attempt from within the vCluster to execute a command on the kubelet proxy will not work and hence make the exploit unusable.

How to try this out yourself

Start by installing the vCluster CLI. Then create a new file called `vcluster.yaml` with:

sync:
  fromHost:
    nodes:
      enabled: true

networking:
  advanced:
    proxyKubelets:
      byIP: true
      byHostname: true

Then create the vCluster via:

vcluster create my-vcluster -f vcluster.yaml

Wait for the vCluster to start and create a new service account, cluster role binding and token within the vCluster:

# Create new service account
kubectl create serviceaccount admin-user -n kube-system

# Create cluster-admin cluster role
kubectl create clusterrolebinding admin-user-binding \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:admin-user

# Create service account token
export TOKEN=$(kubectl create token admin-user -n kube-system)

Then try to execute a command in one of the host cluster pods via:

NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[0].address}')
POD_NAMESPACE=default
POD_NAME=my-host-pod
CONTAINER_NAME=my-container

curl -v -sk -X POST \
  -H "Authorization: Bearer $TOKEN" \
"https://$APISERVER/api/v1/nodes/$NODE_NAME/proxy/exec/$POD_NAMESPACE/$POD_NAME/$CONTAINER_NAME?command=hostname&stdout=true&output=1"

websocat \
  --insecure \
  --header "Authorization: Bearer $TOKEN" \
  --protocol "v4.channel.k8s.io" \
"wss://$NODE_IP:10250/exec/$POD_NAMESPACE/$POD_NAME/$CONTAINER_NAME?output=1&error=1&command=hostname"

This should print the following within the vCluster:

websocat: WebSocketError: I/O failure
websocat: error running

When you try this on the host cluster instead it should work as expected:

nginx
{"metadata":{},"status":"Success"}

This proves that the websocket exploit does not function within the vCluster because it blocks the websocket request altogether when proxying.