Metrics server

Limited vCluster Tenancy Configuration Support

This feature is only available when using the following worker node types:

Host Nodes

Installing metrics server (inside vCluster)

In case the above recommended method of getting metrics in vCluster using the metrics server proxy does not fulfill your requirements and you need a dedicated metrics server installation in the vCluster you can follow this section. Make sure the vCluster has access to the host clusters nodes.

Install the metrics server via the official method into the vCluster.

Wait until the metrics server has started. You should be now able to use kubectl top pods and kubectl top nodes within the vCluster:

kubectl top pods --all-namespaces
NAMESPACE     NAME                              CPU(cores)   MEMORY(bytes)
kube-system   coredns-854c77959c-q5878          3m           17Mi
kube-system   metrics-server-5fbdc54f8c-fgrqk   0m           6Mi

K3s Errors

If you see the below error after installing metrics-server (check k3s#5334 for more information):

loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503

Create a file named metrics_patch.yaml with the following contents:

spec:
  template:
    spec:
      containers:
        - name: metrics-server
          command:
            - /metrics-server
            - --metric-resolution=30s
            - --kubelet-insecure-tls=true
            - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

and apply the patch with kubectl:

kubectl patch deployment metrics-server --patch-file metrics_patch.yaml -n kube-system

How does it work?

By default, vCluster will create a service for each node that redirects incoming traffic from within the vCluster to the node kubelet to vCluster itself. This means that if workloads within the vCluster try to scrape node metrics the traffic reaches vCluster first. Vcluster will redirect the incoming request to the host cluster, rewrite the response (pod names, pod namespaces etc) and return it to the requester.

Reusing host metrics server

Create a vcluster.yaml with:

integrations:
  metricsServer:
    enabled: true
    nodes: true
    pods: true

Then create a vCluster with:

vcluster create host-metrics-vcluster -f vcluster.yaml

Then the metrics api should be available within the vCluster. This will obviously require a metrics server to be installed on the host cluster.

If your metrics server is not using the default kube-system/metrics-server service, you can also configure a custom one via:

integrations:
  metricsServer:
    enabled: true
    apiService:
      service:
        name: my-custom-metrics-server
        namespace: my-custom-namespace
        port: 443

If you are using network policies, you need to add the metrics server pod to the allowed rules the control plane can communicate with:

policies:
  networkPolicy:
    extraControlPlaneRules:
    - to:
        - podSelector:
            matchLabels:
              k8s-app: metrics-server
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: 'my-metrics-server-namespace'

Config reference

metricsServer required object

MetricsServer reuses the metrics server from the host cluster within the vCluster.

enabled required boolean false

Enabled signals the metrics server integration should be enabled.

apiService required object

APIService holds information about where to find the metrics-server service. Defaults to metrics-server/kube-system.

service required object

Service is a reference to the service for the API server.

name required string

Name is the name of the host service of the apiservice.

namespace required string

Namespace is the name of the host service of the apiservice.

port required integer

Port is the target port on the host service to connect to.

nodes required boolean true

Nodes defines if metrics-server nodes api should get proxied from host to virtual cluster.

pods required boolean true

Pods defines if metrics-server pods api should get proxied from host to virtual cluster.