Configure collectors manually
Configure your own OpenTelemetry Collectors when you can't use the bundled cluster-collector
template, such as on vCluster Standalone clusters, or to customize behavior the template
doesn't expose as a parameter. Each collector request must include a bearer token and scope
headers that match a metrics-writer access key. See
Configure metrics access to create the key.
Gateway endpoints​
Inside the Platform cluster, the fleet gateway Service is:
gateway-fleet-observability.vcluster-platform.svc.cluster.local
This endpoint assumes the example connector ID fleet-observability. For another ID,
replace the Service name with gateway-<connector-id>.
A collector inside a cluster resolves names through that cluster's own CoreDNS, so this name only resolves after you replicate the gateway Service into the cluster. See Use a ClusterIP gateway with clusters on the Platform cluster.
Use this OTLP endpoint:
| Protocol | Endpoint |
|---|---|
| OTLP HTTP | https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 |
The examples on this page use OTLP HTTP on port 4318. The gateway also accepts OTLP gRPC
on port 4317, but the bundled OpenTelemetry Collector sends over HTTP, so use 4318.
The gateway always uses TLS. The OpenTelemetry Collector otlp_http exporter accepts the
base URL shown above. Direct OTLP HTTP clients send metrics to the /v1/metrics path on
that host. Mount the CA that issued the gateway serving certificate into the collector and
set the exporter tls.ca_file. The default Platform-managed certificate stores its trust
anchor in ca.crt in the gateway-<connector-id>-tls Secret.
Collectors outside the Platform cluster can't resolve the in-cluster Service DNS name.
Expose the gateway with a LoadBalancer, then use the LoadBalancer address in the exporter
endpoint instead of the Service name, for example
https://<loadbalancer-address>:4318. See
Expose the gateway with a LoadBalancer
to enable it and retrieve the address. The exporter endpoint and TLS settings in the
examples below are otherwise unchanged; only the host differs.
Store the writer key​
Create a Secret for the collector, using a metrics-writer access key from
Create a writer key for a cluster
or Create a writer key for a control plane cluster:
apiVersion: v1
kind: Secret
metadata:
name: metrics-writer-token
namespace: observability
type: Opaque
stringData:
token: REPLACE_WITH_METRICS_WRITER_KEY
Mount or project this Secret into your collector pod and expose it as an environment variable or file.
Cluster collector​
Use tenant scope headers when a collector writes metrics for a specific cluster.
exporters:
otlp_http/fleet:
endpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318
tls:
ca_file: /etc/otel-gateway-tls/ca.crt
headers:
Authorization: Bearer ${env:METRICS_WRITER_TOKEN}
X-Vcluster-Platform-Project: my-project
X-Vcluster-Platform-Instance: my-vcluster
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [otlp_http/fleet]
The metrics-writer key must include a matching spec.scope.virtualClusters entry.
Control plane cluster collector​
Use cluster scope headers when a collector writes metrics for a control plane cluster.
exporters:
otlp_http/fleet:
endpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318
tls:
ca_file: /etc/otel-gateway-tls/ca.crt
headers:
Authorization: Bearer ${env:METRICS_WRITER_TOKEN}
X-Vcluster-Platform-Cluster: local-cluster
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [otlp_http/fleet]
The metrics-writer key must include a matching spec.scope.clusters entry.
Scrape custom targets​
Use the collector's prometheus receiver to scrape exporters such as kubelet,
cAdvisor, DCGM exporter, or application endpoints. The gateway doesn't require a
specific scrape receiver. It only requires OTLP metrics with valid authentication and
scope.
Example DCGM scrape job:
receivers:
prometheus:
config:
scrape_configs:
- job_name: dcgm-exporter
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
action: keep
regex: dcgm-exporter
- source_labels: [__meta_kubernetes_pod_ip]
target_label: __address__
replacement: ${1}:9400
Protect platform labels​
Don't set labels with the vcluster_platform_* prefix in edge collectors. The Write
Gateway strips this namespace from incoming resource, scope, and datapoint attributes
before stamping authenticated labels.
Use your own label prefix for application or infrastructure dimensions.
To check that writes reach the gateway, see Verify writes.