Skip to main content
Version: v4.9 Stable

Prometheus node metrics on shared nodes

In the Shared Nodes tenancy model, vCluster virtualizes node IP addresses by default as part of its kubelet proxying feature. Prometheus requires specific configuration to scrape kubelet and cAdvisor metrics in this environment.

This does not affect Private Nodes tenant clusters, which have dedicated real nodes with directly reachable IPs.

How it works​

vCluster enables kubelet proxying by default (networking.advanced.proxyKubelets.byIP: true and byHostname: true in the Helm values). This feature lets vCluster intercept kubelet traffic and rewrite pod names and namespaces from host-cluster values to their virtual equivalents.

As part of this, vCluster assigns each synced node a virtual identity:

  • With byIP: true, vCluster replaces the node's IP address with the ClusterIP of a per-node Kubernetes Service in the vCluster namespace on the control plane cluster.
  • With byHostname: true, vCluster replaces the node hostname with a synthetic name like <node-name>.nodes.vcluster.com.

Prometheus discovers scrape targets by reading node objects from the tenant cluster's Kubernetes API. It then attempts to connect to those virtual addresses. If the per-node service ClusterIP is not routable from where Prometheus runs, the scrape times out. This most commonly happens when Prometheus is outside the tenant cluster or on a separate connected cluster,

Approaches​

The right approach depends on where Prometheus runs:

  • Prometheus outside the tenant cluster (on the control plane cluster or a central monitoring instance): use Option 1.
  • Prometheus inside the tenant cluster: use Option 2.
  • Specific requirements that prevent Options 1 or 2: use Option 3.

Deploy an OpenTelemetry Collector on the Control Plane Cluster. The collector scrapes cAdvisor and kubelet metrics using the control plane cluster's real node IPs, then pushes enriched metrics to a central Prometheus using remote_write. Because the collector operates at the control plane cluster level, it never reads tenant cluster node objects or depends on virtual node IPs.

This approach also adds per-tenant identity labels such as vcluster_name and vcluster_project to every metric, making it suitable for multi-tenant observability.

See Fleet monitoring with OpenTelemetry for a complete setup guide covering both Shared Nodes and Private Nodes tenancy models.

Option 2: API server proxy scraping (Prometheus inside the tenant cluster)​

Configure Prometheus to route kubelet and cAdvisor scrapes through the Kubernetes API server proxy instead of connecting to node IPs directly. vCluster intercepts requests that go through the API server, forwards them to the real kubelet on the control plane cluster, and returns them with pod names and namespaces already rewritten to their virtual equivalents, so no additional relabeling is needed.

Configure the cAdvisor and kubelet jobs to target the API server and set the metrics path to the node proxy endpoint:

Prometheus scrape config using the API server proxy
scrape_configs:
  - job_name: cadvisor
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

  - job_name: kubelet
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics

Option 3: Disable kubelet proxying​

Setting byHostname and byIP to false causes vCluster to pass through real node addresses from the control plane cluster unchanged. Prometheus can then discover and scrape them directly.

vcluster.yaml
networking:
  advanced:
    proxyKubelets:
      byHostname: false
      byIP: false
info

With kubelet proxying disabled:

  • vCluster does not rewrite metrics. Pod and namespace labels reflect the control plane cluster's internal naming, not the tenant cluster's virtual names. On a shared node, metrics from different tenant clusters will have overlapping pod/namespace label values.
  • Real node IPs are visible in the tenant cluster's node objects, which reveals control plane network topology to tenant cluster users.