Skip to main content

Monitoring & Metrics API

You can monitor the vcluster either from the host cluster or directly from within the vcluster.

info

In order to get node metrics from within the vcluster, vcluster will need to have RBAC permissions to access them. These permissions are given to vcluster when synchronization of the real nodes is enabled. See Nodes documentation page for more details.

Installing metrics server#

Make sure the vcluster has access to the host clusters nodes. Enabling real nodes synchronization will create the required RBAC permissions.

Install the metrics server via the official method into the vcluster.

Wait until the metrics server has started. You should be now able to use kubectl top pods and kubectl top nodes within the vcluster:

kubectl top pods --all-namespaces
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system coredns-854c77959c-q5878 3m 17Mi
kube-system metrics-server-5fbdc54f8c-fgrqk 0m 6Mi

If you see below error after installing metrics-server (check k3s#5334 for more information):

loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503

Create a file named 'metrics_patch.yaml' with the following contents:

spec:
template:
spec:
containers:
- name: metrics-server
command:
- /metrics-server
- --metric-resolution=30s
- --kubelet-insecure-tls=true
- --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

and apply the patch with kubectl:

kubectl patch deployment metrics-server --patch-file metrics_patch.yaml -n kube-system

How does it work?#

By default, vcluster will create a service for each node which redirects incoming traffic from within the vcluster to the node kubelet to vcluster itself. This means that if workloads within the vcluster try to scrape node metrics the traffic reaches vcluster first. Vcluster will redirect the incoming request to the host cluster and rewrite the response (pod names, pod namespaces etc) and return it to the requester.

Monitoring the vcluster#

Vcluster is able to rewrite node stats and metrics. This means monitoring a vcluster works similar to monitoring a regular Kubernetes cluster.

info

You need to make sure that vcluster has access to the host clusters nodes. Enabling real nodes synchronization will create the required RBAC permissions.

Please follow the official Kuberentes documentation on how to monitor a Kubernetes cluster.

How does it work?#

By default, vcluster will create a service for each node which redirects incoming traffic from within the vcluster to the node kubelet to vcluster itself. This means that if workloads within the vcluster try to scrape node metrics the traffic reaches vcluster first. Vcluster will redirect the incoming request to the host cluster and rewrite the response (pod names, pod namespaces etc) and return it to the requester.

Monitoring the vcluster StatefulSet#

vcluster exposes metrics endpoints on https://0.0.0.0:8443/metrics (syncer metrics) and https://0.0.0.0:6444/metrics (k3s metrics). In order to scrape those metrics, you will need to send an Authorization header with a valid virtual cluster service account token, that has permissions to access the /metrics endpoint within the vcluster.