Skip to main content
Version: v5.0 Stable

Configure gateway networking and TLS

The observability gateway serves OTLP writes and PromQL reads over TLS, and connects to the metrics backend over TLS or mTLS. After you install the gateway, use this page to bring your own serving certificate, expose the gateway to collectors outside the Platform cluster, or authenticate the gateway to the backend with mTLS.

Certificate changes restart the gateway

Changing the contents of the gateway serving certificate Secret or the metrics backend certificate Secret restarts the gateway pod. Telemetry the gateway accepted but hasn't delivered yet is lost on restart, because its queues are held in memory only. Rotate certificates while the metrics backend and every destination are reachable. See When the gateway restarts.

Customize the gateway serving certificate​

The gateway always serves OTLP and query traffic over TLS. When you don't specify a certificate Secret, Platform creates and manages a self-signed Secret named gateway-<connector-id>-tls. This only works for connector IDs of 51 characters or fewer; see Create the connector Secret for why longer IDs need a custom certificate Secret instead.

To use your own certificate, create a Secret in the Platform namespace. Platform never validates or modifies a custom certificate, so it must already cover every hostname a client connects to. At minimum, include the gateway's in-cluster Service hostname, gateway-<connector-id>.<platform-namespace>.svc.cluster.local, since Platform-generated Grafana datasources and any in-cluster collector always connect through it. If you expose the gateway with a LoadBalancer (see Expose the gateway with a LoadBalancer below), also include its external hostname or IP as a SAN.

gateway-tls.yaml
apiVersion: v1
kind: Secret
metadata:
name: fleet-gateway-tls
namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
tls.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Reference it from the connector:

stringData:
observabilityGatewayCertSecretName: fleet-gateway-tls

Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom certificate cases. Distribute ca.crt to collectors that don't already trust the issuing CA. Grafana datasources generated by Platform trust the ca.crt from the gateway serving certificate Secret. A certificate signed by a public CA still needs to cover the hostnames above; once it does, ca.crt can be omitted, since clients already trust the public CA through their system trust stores.

Expose the gateway with a LoadBalancer​

The gateway Service defaults to ClusterIP. To make the gateway reachable from collectors and query clients outside the Platform cluster, set serviceType to LoadBalancer. You can optionally apply provider-specific Service annotations:

stringData:
serviceType: LoadBalancer
serviceAnnotations: |
service.beta.kubernetes.io/aws-load-balancer-scheme: internal

For a Platform-managed certificate, the controller waits for the LoadBalancer address and adds it to the certificate as a subject alternative name before deploying the gateway. A custom certificate needs that same SAN added by hand, since Platform never modifies it. See Customize the gateway serving certificate for how clients trust the resulting certificate either way.

Which clusters need a LoadBalancer gateway​

A ClusterIP gateway is only reachable from inside the Platform cluster. It can serve a cluster whose control plane runs on the Platform cluster and that doesn't use private nodes, such as a cluster with shared nodes, because that cluster's collector runs as a pod on a Platform cluster node. Such a cluster still needs one extra step to resolve the gateway; see Use a ClusterIP gateway with clusters on the Platform cluster.

Every other cluster runs its collector where a ClusterIP gateway is unreachable:

  • A cluster hosted on a connected control plane cluster runs its collector on that connected cluster's nodes.
  • A cluster with private nodes runs its collector on its own nodes, even when its control plane runs on the Platform cluster.
  • A vCluster Standalone cluster runs on its own machine.

Rather than hand out an address those clusters can't reach, Platform refuses them the metrics-writer credential for a ClusterIP connector, so no credential Secret is delivered. Platform also refuses a cluster whose status it can't read yet. Set serviceType: LoadBalancer on the connector to serve these clusters.

Retrieve the load balancer address​

The cloud provider assigns the address asynchronously, so it stays empty until provisioning finishes. Read it from the gateway Service:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \
-o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}'

The provider reports either a hostname (for example, AWS) or an IP address (for example, GCP and Azure). Build the reachable endpoints from that address:

EndpointPortPurpose
https://<loadbalancer-address>:43184318OTLP HTTP writes. Use this for collectors.
https://<loadbalancer-address>:43174317OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use 4318 for collectors.
https://<loadbalancer-address>:80818081Prometheus HTTP API through the Query Proxy. Use this for external query clients.

Use this address wherever a collector or query client connects from outside the Platform cluster.

Use a ClusterIP gateway with clusters on the Platform cluster​

A ClusterIP gateway suits evaluation and single-cluster installs where every cluster runs on the Platform cluster without private nodes. Those clusters' collectors can reach the gateway Service over the network, but can't resolve its name by default. A collector resolves names through its cluster's own CoreDNS, which only answers for the cluster's own Services, so the delivered endpoint gateway-<connector-id>.<platform-namespace>.svc.cluster.local doesn't resolve.

Replicate the gateway Service into each such cluster under the same namespace and name. The delivered hostname then resolves and still matches the gateway serving certificate:

Modify the following with your specific values to generate a copyable command:
vcluster.yaml
networking:
replicateServices:
fromHost:
- from: vcluster-platform/gateway-fleet-observability
to: vcluster-platform/gateway-fleet-observability

vCluster creates the target namespace in the cluster if it's missing. See Replicate services for how replication works.

An install where the Platform cluster only hosts clusters configured this way needs no load balancer.

Use backend mTLS​

To authenticate the gateway to your backend with mTLS, create a Secret:

backend-client-tls.yaml
apiVersion: v1
kind: Secret
metadata:
name: metrics-backend-client-tls
namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----

Reference it from the connector:

stringData:
metricsBackendCertSecretName: metrics-backend-client-tls

Configure backend mTLS across namespaces or clusters​

The bundled Prometheus template's default mTLS setup works because Prometheus and the gateway both mount the same Secret object. By default they share the namespace Platform itself is installed in (vcluster-platform, unless you renamed it), so metricsBackendTLSSecret (the Prometheus side) and metricsBackendCertSecretName (the connector side) both point at the same auto-generated metrics-backend-mtls Secret.

That single-Secret shortcut only works when Prometheus and the gateway share a namespace and a cluster. What to do instead depends on where Prometheus runs.

Different namespace, same cluster​

If you set destinationNamespace to a namespace other than the one Platform is installed in, Prometheus stays on the local cluster, so its bootstrap certificate still covers the right in-cluster hostname (prom-prometheus-server.<destinationNamespace>.svc.cluster.local). A single Secret object just can't be mounted in two namespaces at once, so copy it instead of sharing one object:

  • Leave metricsBackendTLSSecret empty so the Argo CD template bootstraps the shared certificate for you, named metrics-backend-mtls in Prometheus's destinationNamespace. See Deploy the metrics backend with Argo CD.
  • Once Prometheus reports ready, read that generated Secret and create an identical copy in the Platform namespace, then reference it from the connector's metricsBackendCertSecretName.

Setting metricsBackendTLSSecret to a name of your own disables this bootstrap entirely, so only use it if you're supplying your own certificate material for both Secrets instead of copying the generated one.

A different cluster, or a backend other than the bundled Prometheus​

The bundled Prometheus workflow deploys Prometheus to the local cluster, and its bootstrap certificate only covers in-cluster Service hostnames. Copying that Secret doesn't work once the gateway reaches the backend through an external hostname, because the certificate has no SAN for it. Create the gateway-side Secret as described in Use backend mTLS above, but with two differences:

  • The backend's server certificate must cover the hostname used in metricsBackendOtlpEndpoint and metricsBackendPromQLEndpoint, not an in-cluster Service name.
  • Configure the backend to trust the CA that issues the gateway's client certificate, the same way the bundled Prometheus config trusts it through client_ca_file. Without this, the backend rejects the gateway's mTLS handshake even if the gateway trusts the backend.