Monitoring & Logging
You can monitor the vcluster either from the host cluster or directly from within the vcluster.
In order to get node metrics from within the vcluster, vcluster will need to have RBAC permissions to access them. These permissions are given to vcluster when synchronization of the real nodes is enabled. See Nodes documentation page for more details.
Enabling the metrics server proxy (Recommended)
This feature requires a working installation of metrics server on the host cluster
Its possible to proxy the metrics server in the underlying host cluster and get the pod
/node
metrics individually or both of them according to the use case. This can be enabled with the following values:
proxy:
metricsServer:
nodes:
enabled: true
pods:
enabled: true
Installing metrics server (inside vcluster)
In case the above recommended method of getting metrics in vcluster using the metrics server proxy does not fulfil your requirements and you need a dedicated metrics server installation in the vcluster you can follow this section. Make sure the vcluster has access to the host clusters nodes. Enabling real nodes synchronization will create the required RBAC permissions.
Install the metrics server via the official method into the vcluster.
Wait until the metrics server has started. You should be now able to use kubectl top pods
and kubectl top nodes
within the vcluster:
kubectl top pods --all-namespaces
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system coredns-854c77959c-q5878 3m 17Mi
kube-system metrics-server-5fbdc54f8c-fgrqk 0m 6Mi
If you see below error after installing metrics-server (check k3s#5334 for more information):
loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503
Create a file named 'metrics_patch.yaml' with the following contents:
spec:
template:
spec:
containers:
- name: metrics-server
command:
- /metrics-server
- --metric-resolution=30s
- --kubelet-insecure-tls=true
- --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
and apply the patch with kubectl:
kubectl patch deployment metrics-server --patch-file metrics_patch.yaml -n kube-system
How does it work?
By default, vcluster will create a service for each node which redirects incoming traffic from within the vcluster to the node kubelet to vcluster itself. This means that if workloads within the vcluster try to scrape node metrics the traffic reaches vcluster first. Vcluster will redirect the incoming request to the host cluster and rewrite the response (pod names, pod namespaces etc) and return it to the requester.
Monitoring the vcluster
Vcluster is able to rewrite node stats and metrics. This means monitoring a vcluster works similar to monitoring a regular Kubernetes cluster.
You need to make sure that vcluster has access to the host clusters nodes. Enabling real nodes synchronization will create the required RBAC permissions.
Please follow the official Kuberentes documentation on how to monitor a Kubernetes cluster.
How does it work?
By default, vcluster will create a service for each node which redirects incoming traffic from within the vcluster to the node kubelet to vcluster itself. This means that if workloads within the vcluster try to scrape node metrics the traffic reaches vcluster first. Vcluster will redirect the incoming request to the host cluster and rewrite the response (pod names, pod namespaces etc) and return it to the requester.
Monitoring the vcluster StatefulSet
vcluster exposes metrics endpoints on https://0.0.0.0:8443/metrics
(syncer metrics) and https://0.0.0.0:6444/metrics
(k3s metrics). In order to scrape those metrics, you will need to send an Authorization
header with a valid virtual cluster service account token, that has permissions to access the /metrics
endpoint within the vcluster.
Logging
You can enable logging for vcluster pods right from host cluster or from within each vcluster as well.
Enabling Hostpath Mapper
Vcluster internal logging relies on enabling a vcluster component called the Hostpath Mapper. This will make sure to resolve the correct virtual pod and container names to their physical counterparts.
To enable this component, simply create or upgrade an existing vcluster with the following values
hostpathMapper:
enabled: true
Once deployed successfully a new Daemonset
component of vcluster would start running on every node.
We can now install our desired logging stack and start collecting the logs.
Logging with ELK and fluentd inside vcluster:
Install the ELK stack:
helm upgrade --install elk-elasticsearch elastic/elasticsearch -f elastic_values.yaml -n logging --create-namespace
helm upgrade --install elk-logstash elastic/logstash -f logstash_values.yaml -n logging
helm upgrade --install elk-kibana elastic/kibana -f kibana_values.yaml -n logging
# optionally install filebeat if you plan to use filebeat instead of fluentd
helm upgrade --install elk-filebeat elastic/filebeat -f filebeat_values.yaml -n loggingNext install fluentd daemonset, this can be found on github:
kubectl apply -f fluentd-daemonset-elasticsearch.yaml
Alternatively, you can also deploy via the helm charts provided by fluentbit.
Check for available indices -
port-forward
theelasticsearch-master
on port9100
and visit the http://localhost:9200/_cat/indices, you should see the followinglogstash-*
indices available:green open .geoip_databases rP6BifVQSuCv1XmctC0M_Q 1 0 40 0 38.4mb 38.4mb
green open .kibana_task_manager_7.17.3_001 p5Idg-xWTpCj4TWh6YpNrQ 1 0 17 543 123.6kb 123.6kb
yellow open logstash-2022.10.10 nyG-OW_qRKCBertmmOwwyw 1 1 895 0 416.6kb 416.6kb ◀─────┐
green open .apm-custom-link jv3jzCztQUujEYwYv1iTIw 1 0 0 0 226b 226b │ ┌───────────────┐
green open .apm-agent-configuration NsZHlaeGSmSc7xSa8CGcOA 1 0 0 0 226b 226b │ │ Logstash │
yellow open logstash-2022.10.07 cW3b1TJlROCwV2BKkzpt2Q 1 1 212 0 52.1kb 52.1kb ◀─────┼──────│ Entries │
yellow open logstash-2022.10.08 yzU4pqq3QOyZkukcmGKpaw 1 1 172 0 43.6kb 43.6kb ◀─────┤ └───────────────┘
yellow open logstash-2022.10.09 n9GQnFB4RSWlWwkFG1848g 1 1 866 0 100.4kb 100.4kb ◀─────┘
green open .kibana_7.17.3_001 BjXjQqXcRoiiGQg_zsrSrg 1 0 21 8 2.3mb 2.3mbNext
port-forward
the kibana dashboard on its default port5601
and navigate to http://localhost:5601/app/management or choose "Stack Management" from left menu side bar.Choose "Index Patterns" and click on "Create index Pattern"
Type the Name as
logstash*
and@timestamp
for the Timestamp field and click on "Create index pattern"Now you can navigate to http://localhost:5601/app/discover or click on "Discover" from the left sidebar menu and should start seeing your logs.
Logging with Grafana and Loki
- Install the Prometheus stack:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack --namespace monitor --create-namespace - Install Loki:
helm repo add loki https://grafana.github.io/loki/charts
helm upgrade --install loki --namespace=monitoring grafana/loki-stack --create-namespace - Open the Grafana Dashboard:
- Port-forward grafana dashboard
kubectl port-forward -n monitor service/prometheus-grafana 3000:80
- Get Grafana credentials
kubectl get secrets -n monitor prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 -D
- Navigate to http://localhost:3000
- Port-forward grafana dashboard
- Add a data source by navigating to http://localhost:3000/datasources or click "Data Sources" under the ⚙️ icon from left menu
- Click on "Add Data Sources" and select "Loki" from the list.
- Enter the loki endpoint in the
URL
field ashttp://loki.monitoring:3100
or to the corresponding<name>.<namespace>:<port>
value according to your deployment, and click on "Save & test". - Next click on "Explore" or navigate to http://localhost:3000/explore and select "Loki" from the dropdown menu. Select the desired Labels and Click on "Run query". Youre logs should now start appearing.