Cilium Powered vClusters - Part 4: Observability with Hubble

Enable Cilium Hubble to visualize traffic across tenant clusters. See allowed and dropped flows in real time, and understand exactly which policy is blocking which connection.

Part 3 proved isolation using curl timeouts. Part 4 makes the policy decisions visible: which flows were dropped, when, and why.

Why Hubble?

In Part 3 we enforced tenant isolation with CiliumNetworkPolicy. We proved it worked because curl timed out (exit code 28), but that does not tell you:

• why it timed out,
• which policy caused the drop,
• or exactly when the drop happened.

Hubble solves this by exposing Cilium’s eBPF-level visibility via:

• a browser UI (graph of connections), and
• a CLI (raw flow records, identities, drop reasons).

What is Hubble?

Hubble has two components:

Neither is enabled by default. You add them on top of an existing Cilium install with a Helm upgrade (no reinstall) and existing policies/connectivity remain intact.

Step 1 - Enable Hubble

Check current Helm values:

helm get values cilium -n kube-system


Upgrade Cilium to enable Hubble (preserve existing values):

helm upgrade cilium cilium/cilium \\
 --version 1.16.0 \\
 --namespace kube-system \\
 --reuse-values \\
 --set hubble.enabled=true \\
 --set hubble.relay.enabled=true \\
 --set hubble.ui.enabled=true


Step 2 - Wait for Hubble to be ready

kubectl rollout status deployment/hubble-relay -n kube-system
kubectl rollout status deployment/hubble-ui -n kube-system


Expected:

• deployment "hubble-relay" successfully rolled out
• deployment "hubble-ui" successfully rolled out

Step 3 - Open Hubble UI

Hubble UI is not exposed externally. Use port-forward:

# Run in a separate terminal and keep it running
kubectl port-forward -n kube-system svc/hubble-ui 12000:80


Open http://localhost:12000 in your browser.

Step 4 - Visualize dropped traffic

1. Apply the CiliumNetworkPolicy from Part 3.
2. Generate cross-tenant traffic from tc-gamma to tc-alpha and tc-beta in parallel:

vcluster connect tc-gamma --namespace tc-gamma --driver helm

kubectl exec curl -- curl -s --max-time 5 http://<TC_ALPHA_NGINX_IP>/ &
kubectl exec curl -- curl -s --max-time 5 http://<TC_BETA_NGINX_IP>/ &
wait


Why parallel (&)? Each curl waits up to 5s before timing out. Parallel execution makes both flows show up in the same Hubble time window.

In Hubble UI:

• Select namespace: tc-gamma
• Verdict filter: Dropped

You should see two red flows (port 80) from tc-gamma to tc-alpha and tc-beta.

Two red lines, one from tc-gamma to tc-alpha on port 80, one from tc-gamma to tc-beta on port 80. Both dropped simultaneously. The dashed box shows the tc-gamma namespace boundary. Any connection leaving that box towards another Tenant Cluster is blocked by the CiliumNetworkPolicy.

Step 5 - Use the Hubble CLI (raw flow data)

Run from the cilium-vind context:

kubectl exec -n kube-system -it ds/cilium -- hubble observe \\
 --namespace tc-gamma \\
 --last 20


Example drop output:

Apr 29 17:57:28.546: tc-gamma/curl-x-default-x-tc-gamma:44140 (ID:56702) <> tc-alpha/nginx-x-default-x-tc-alpha:80 (ID:9833) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Apr 29 17:57:28.546: tc-gamma/curl-x-default-x-tc-gamma:44140 (ID:56702) <> tc-alpha/nginx-x-default-x-tc-alpha:80 (ID:9833) Policy denied DROPPED (TCP Flags: SYN)


How to read it:

‍

You can also view from the receiver side:

kubectl exec -n kube-system -it ds/cilium -- hubble observe \\
 --namespace tc-alpha \\
 --last 20


Key takeaway

With standard NetworkPolicy, you often only see “silence” when traffic is blocked. With CiliumNetworkPolicy + Hubble, you get complete visibility: every flow, every drop, pod identities, drop reason, and timestamp.

We now have a fully observable Cilium-powered multi-tenant setup: three Tenant Clusters with enforced isolation and real-time traffic visibility.

References (Part 4)

• Hubble network observability documentation
• Setting up Hubble observability

‍