Version: main 🚧

Install the observability gateway

The observability gateway runs in the vCluster Platform namespace. It contains two containers:

otel-gateway, the Write Gateway that receives OTLP.
query-proxy, the read proxy that serves a Prometheus-compatible query API for PromQL reads.

Platform deploys the gateway from an observability connector Secret. The connector stores the metrics backend endpoints and optional TLS settings.

Prerequisites

Before you create the connector, prepare:

A metrics backend that ingests OTLP metrics and serves the Prometheus HTTP API for queries. Deploy the bundled Prometheus out of the box with the Argo CD integration, or bring your own.
Network access from the Platform namespace to both backend endpoints.
A vCluster Platform license that includes Fleet Observability.
Optional custom gateway serving certificate or backend mTLS Secrets.

note

The observability gateway doesn't install or size the metrics backend for you. Bring your own backend, or deploy the bundled Prometheus backend out of the box with the Argo CD integration. See Deploy Fleet Observability with Argo CD.

Create the connector Secret

Create the Fleet Observability connector in the Platform namespace. The connector ID can be any valid Secret name. Platform derives the gateway resource name as gateway-<connector-id>. The examples use fleet-observability, which produces gateway-fleet-observability.

The loft.sh/connector-type: observability label is required. Platform only recognizes and reconciles a Secret as an observability connector when this label is present.

Mark the connector as the fleet default with the platform.vcluster.com/fleet-observability-connector label set to "true". Only one connector can be the fleet default.

Platform UI
YAML

fleet-observability-connector.yaml
apiVersion: v1
kind: Secret
metadata:
  name: fleet-observability
  namespace: vcluster-platform
  labels:
    loft.sh/connector-type: observability
    platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
  metricsBackendOtlpEndpoint: https://metrics.example.com/otlp
  metricsBackendPromQLEndpoint: https://metrics.example.com

Apply the Secret:

kubectl apply -f fleet-observability-connector.yaml

Click Connectors and select the Observability tab.
Click .
In the Display name field, enter a human-readable name for the connector. The Observability Connector ID is auto-generated from the display name. The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default.
In the Metrics Backend OTLP Endpoint field, enter the OTLP endpoint that receives metrics from the Write Gateway.
In the Metrics Backend PromQL Endpoint field, enter the Prometheus HTTP API endpoint queried by the Query Proxy.
Optionally configure TLS for the metrics backend and the gateway. See Connector keys for the available settings.
Click .

Platform reconciles the following resources:

Deployment/gateway-fleet-observability
Service/gateway-fleet-observability
Grafana datasource ConfigMaps for fleet, control plane cluster, and tenant cluster views when Grafana datasource reconciliation is enabled.

Connector keys

Key	Required	Description
`metricsBackendOtlpEndpoint`	Yes	OTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends `/v1/metrics` (for example, `https://prom-prometheus-server.vcluster-platform/api/v1/otlp` receives writes at `/api/v1/otlp/v1/metrics`).
`metricsBackendPromQLEndpoint`	Yes	Prometheus HTTP API endpoint queried by the Query Proxy.
`metricsBackendInsecureSkipVerify`	No	Set to `"true"` only for test backends with unverifiable TLS.
`metricsBackendServerName`	No	TLS server name override for the metrics backend.
`observabilityGatewayCertSecretName`	No	Custom Secret with `tls.crt` and `tls.key` for gateway serving TLS. Include `ca.crt` when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted.
`metricsBackendCertSecretName`	No	Secret with `ca.crt`, `tls.crt`, and `tls.key` for mTLS to the backend. The gateway trusts the backend with `ca.crt` and presents the certificate and key as its client identity.
`gatewayResources`	No	YAML `resources` block for the Write Gateway container.
`queryProxyResources`	No	YAML `resources` block for the Query Proxy container.
`serviceType`	No	Gateway Service type. Valid values are `ClusterIP`, the default, and `LoadBalancer`.
`serviceAnnotations`	No	YAML map of annotations applied to a `LoadBalancer` gateway Service.
`grafanaUrl`	No	Grafana upstream URL used by the Platform `/grafana/` reverse proxy.
`grafanaInsecureSkipVerify`	No	Set to `"true"` only to test a Grafana endpoint with an unverifiable certificate.
`grafanaCertSecretName`	No	Grafana serving certificate Secret. Platform uses its `ca.crt` to trust the Grafana upstream.
`fleetObservabilityDashboardUid`	No	UID of the fleet dashboard embedded by the Fleet Observability page.

Customize the gateway serving certificate

The gateway always serves OTLP and query traffic over TLS. When you don't specify a certificate Secret, Platform creates and manages a self-signed Secret named gateway-<connector-id>-tls.

To use your own certificate, create a Secret in the Platform namespace:

gateway-tls.yaml
apiVersion: v1
kind: Secret
metadata:
  name: fleet-gateway-tls
  namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Reference it from the connector:

stringData:
  observabilityGatewayCertSecretName: fleet-gateway-tls

Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom certificate cases. Distribute ca.crt to collectors that don't already trust the issuing CA. Grafana datasources generated by Platform trust the ca.crt from the gateway serving certificate Secret. If a custom certificate is signed by a public CA, ca.crt can be omitted and clients use their system trust stores.

Expose the gateway with a LoadBalancer

The gateway Service defaults to ClusterIP. To make the gateway reachable from collectors and query clients outside the Platform cluster, set serviceType to LoadBalancer. You can optionally apply provider-specific Service annotations:

stringData:
  serviceType: LoadBalancer
  serviceAnnotations: |
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal

For a Platform-managed certificate, the controller waits for the LoadBalancer address and adds it to the certificate as a subject alternative name before deploying the gateway. This means clients that verify TLS trust the address without extra configuration. For a self-signed certificate, distribute ca.crt as described in Customize the gateway serving certificate. A certificate signed by a public CA needs no extra trust configuration.

Retrieve the load balancer address

The cloud provider assigns the address asynchronously, so it stays empty until provisioning finishes. Read it from the gateway Service:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \
  -o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}'

The provider reports either a hostname (for example, AWS) or an IP address (for example, GCP and Azure). Build the reachable endpoints from that address:

Endpoint	Port	Purpose
`https://<loadbalancer-address>:4318`	`4318`	OTLP HTTP writes. Use this for collectors.
`https://<loadbalancer-address>:4317`	`4317`	OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use `4318` for collectors.
`https://<loadbalancer-address>:8081`	`8081`	Prometheus HTTP API through the Query Proxy. Use this for external query clients.

Use this address wherever a collector or query client connects from outside the Platform cluster.

Use backend mTLS

To authenticate the gateway to your backend with mTLS, create a Secret:

backend-client-tls.yaml
apiVersion: v1
kind: Secret
metadata:
  name: metrics-backend-client-tls
  namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----

Reference it from the connector:

stringData:
  metricsBackendCertSecretName: metrics-backend-client-tls

Verify the gateway

Check that the gateway pod is running:

CONNECTOR_ID=fleet-observability
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID"

Check the service ports:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID"

Expected service ports:

Port	Name	Purpose
`4317`	`otlp`	OTLP gRPC writes.
`4318`	`otlp-http`	OTLP HTTP writes.
`8081`	`query`	Prometheus HTTP API through the Query Proxy.

Next, create metrics access keys in Configure metrics access.

Prerequisites​

Create the connector Secret​

Connector keys​

Customize the gateway serving certificate​

Expose the gateway with a LoadBalancer​

Retrieve the load balancer address​

Use backend mTLS​

Verify the gateway​