Share Istio with vCluster from the host cluster
Sharing an Istio installation between a host cluster and vCluster is a practical approach for ingress, but comes with important limitations:
- Istio sidecar containers cannot run within vCluster.
- Service-to-service mesh communication isn't supported.
- Advanced mesh features like mutual TLS and traffic policies are unavailable.
You can, however, sync Istio Gateway and VirtualService resources from the vCluster to the host cluster.
Software versions​
This guide uses the following versions:
- vCluster v0.21.0
- Kubernetes v1.30.5
- Istio v1.22.3
- Kubernetes Gateway API v1.1
Configure Kubernetes Gateway API sync​
Use this configuration to sync Gateway API resources. This works with both HTTPRoute and GRPCRoute custom resources.
sync:
toHost:
ingresses:
enabled: true
customResources:
gateways.gateway.networking.k8s.io:
enabled: true
httproutes.gateway.networking.k8s.io:
enabled: true
patches:
- path: .spec.parentRefs[*].name
reference:
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
- path: .spec.rules[*].backendRefs[*].name
reference:
kind: Service
apiVersion: v1
Configure Istio Gateway and VirtualService sync​
Use this configuration to sync Istio-specific resources:
sync:
toHost:
ingresses:
enabled: true
customResources:
gateways.networking.istio.io:
enabled: true
virtualservices.networking.istio.io:
enabled: true
patches:
- path: .spec.gateways[*]
reference:
kind: Gateway
apiVersion: networking.istio.io/v1beta1
- path: .spec.http[*].route[*].destination.host
reference:
kind: Service
apiVersion: v1
Set up generic sync for Istio resources​
For more advanced scenarios, use generic sync:
experimental:
genericSync:
role:
extraRules:
- apiGroups: ["networking.istio.io"]
resources: ["virtualservices", "destinationrules", "serviceentries", "gateways"]
verbs: ["create", "delete", "patch", "update", "get", "list", "watch"]
# extra cluster scoper permissions required for the plugin
clusterRole:
extraRules:
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch"]
export:
- apiVersion: networking.istio.io/v1
kind: Gateway
- apiVersion: networking.istio.io/v1
kind: VirtualService
patches:
- op: rewriteName
path: .spec.gateways[*]
regex: "($NAMESPACE/)?$NAME"
conditions:
- notEqual: "mesh"
- op: rewriteName
path: .spec.http[*].route[*].destination.host
regex: &svcHost >
^$NAME((\.$NAMESPACE)?(\.svc(\.cluster\.local)?){1})?$
reversePatches:
- op: copyFromObject
fromPath: status
path: status
Recognize sidecar injection issues​
When Istio sidecar injection is enabled (on the namespace or pod), the following errors can occur:
-
The istio-proxy container displays this error and never reaches
Runningstatus:2024-10-24T15:52:58.785827Z warn xdsproxy upstream terminated with unexpected error rpc error: code = PermissionDenied desc = authorization failed: no identities ([spiffe://cluster.local/ns/vcluster-abbea20e-2b732699/sa/vc-workload-istio-test-24]) matched httpbin/httpbin id=14 -
The
istio-proxycontainer sets two environment variables with vCluster values:- name: istio-proxy
image: docker.io/istio/proxyv2:1.22.3
args:
- proxy
- sidecar
- '--domain'
- $(POD_NAMESPACE).svc.cluster.local
...
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
Full examples​
Istio sidecar on namespace with multi-namespace mode​
apiVersion: management.loft.sh/v1
kind: App
metadata:
name: istio-base
spec:
displayName: Istio Base
owner:
team: loft-admins
recommendedApp:
- cluster
defaultNamespace: istio-system
icon: <https://seeklogo.com/images/I/istio-logo-92FF583709-seeklogo.com.png>
config:
chart:
name: base
version: 1.22.3
repoURL: <https://istio-release.storage.googleapis.com/charts>
---
apiVersion: management.loft.sh/v1
kind: App
metadata:
name: istiod
spec:
displayName: Istiod
owner:
team: loft-admins
recommendedApp:
- cluster
defaultNamespace: istio-system
icon: <https://seeklogo.com/images/I/istio-logo-92FF583709-seeklogo.com.png>
config:
chart:
name: istiod
version: 1.22.3
repoURL: <https://istio-release.storage.googleapis.com/charts>
---
apiVersion: management.loft.sh/v1
kind: App
metadata:
name: istio-ingress
spec:
displayName: Istio Ingress Gateway
owner:
team: loft-admins
recommendedApp:
- cluster
defaultNamespace: istio-ingress
icon: <https://seeklogo.com/images/I/istio-logo-92FF583709-seeklogo.com.png>
config:
chart:
name: gateway
version: 1.22.3
repoURL: <https://istio-release.storage.googleapis.com/charts>
---
apiVersion: management.loft.sh/v1
kind: VirtualClusterTemplate
metadata:
name: istio-virtual-service-example
spec:
displayName: Istio Virtual Service Example
description: >-
This virtual cluster template deploys a vCluster with generic sync configured to sync `virtualservices.networking.istio.io` from the vCluster to the host.
owner:
team: loft-admins
template:
metadata: {}
instanceTemplate:
metadata: {}
pro: {}
helmRelease:
chart:
version: 0.20.0-beta.15
values: |-
sync:
toHost:
ingresses:
enabled: true
# Checkout <https://vcluster.com/pro/docs/> for more configuration options
accessPoint:
ingress: {}
spaceTemplate:
metadata: {}
versions:
- template:
metadata:
annotations:
sleepmode.loft.sh/ignore-user-agents: argo*
instanceTemplate:
metadata:
labels:
env: '{{ .Values.env }}'
team: '{{ .Values.loft.project }}'
objects: |-
apiVersion: v1
kind: Namespace
metadata:
name: httpbin
labels:
istio-injection: enabled
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin
namespace: httpbin
---
apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: httpbin
labels:
app: httpbin
service: httpbin
spec:
ports:
- name: http
port: 8000
targetPort: 8080
selector:
app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin
namespace: httpbin
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v1
template:
metadata:
labels:
app: httpbin
version: v1
spec:
serviceAccountName: httpbin
containers:
- image: docker.io/kong/httpbin:0.1.0
imagePullPolicy: IfNotPresent
name: httpbin
# Same as found in Dockerfile's CMD but using an unprivileged port
command:
- gunicorn
- -b
- "[::]:8080"
- httpbin:app
- -k
- gevent
env:
# Tells pipenv to use a writable directory instead of $HOME
- name: WORKON_HOME
value: /tmp
ports:
- containerPort: 8080
---
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
name: httpbin-gateway
namespace: httpbin
spec:
# The selector matches the ingress gateway pod labels.
# If you installed Istio using Helm following the standard documentation, this would be "istio=ingress"
selector:
istio: ingress
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: httpbin
namespace: httpbin
spec:
hosts:
- "*"
gateways:
- httpbin-gateway
http:
- match:
- uri:
prefix: /headers
route:
- destination:
port:
number: 8000
host: httpbin
pro:
enabled: true
helmRelease:
chart:
version: 0.21.0-beta.5
values: |-
external:
platform:
autoSleep:
afterInactivity: 6000
autoDelete:
afterInactivity: 150000
sync:
toHost:
ingresses:
enabled: true
serviceAccounts:
enabled: true
networking:
advanced:
fallbackHostCluster: true
experimental:
multiNamespaceMode:
enabled: true
genericSync:
role:
extraRules:
- apiGroups: ["networking.istio.io"]
resources: ["virtualservices", "destinationrules", "serviceentries", "gateways"]
verbs: ["create", "delete", "patch", "update", "get", "list", "watch"]
# extra cluster scoper permissions required for the plugin
clusterRole:
extraRules:
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch"]
export:
- apiVersion: networking.istio.io/v1
kind: Gateway
- apiVersion: networking.istio.io/v1
kind: VirtualService
patches:
- op: rewriteName
path: .spec.gateways[*]
regex: "($NAMESPACE/)?$NAME"
conditions:
- notEqual: "mesh"
- apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
patches:
- op: rewriteName
path: .spec.exportTo
regex: $NAMESPACE
conditions:
- notEqual: "."
import:
- kind: Secret
apiVersion: v1
controlPlane:
statefulSet:
labels:
env: '{{ .Values.env }}'
team: '{{ .Values.loft.project }}'
# Use an embedded managed etcd server instead of using the default SQLite backend
backingStore:
etcd:
embedded:
enabled: true
distro:
k8s:
apiServer:
image:
tag: "{{ .Values.k8sVersion }}"
controllerManager:
image:
tag: "{{ .Values.k8sVersion }}"
coredns:
embedded: true
# Checkout <https://vcluster.com/pro/docs/> for more config options
accessPoint:
ingress: {}
spaceTemplate:
metadata: {}
parameters:
- variable: k8sVersion
label: k8sVersion
description: Please select Kubernetes version
options:
- v1.30.2
- v1.29.6
- v1.28.11
- v1.27.9
- v1.26.15
defaultValue: v1.29.6
- variable: env
label: Deployment Environment
description: >-
Environment for deployments for this vCluster used as cluster label
for Argo CD ApplicationSet Cluster Generator
options:
- dev
- qa
- prod
defaultValue: dev
version: 1.0.0
- template:
metadata: {}
instanceTemplate:
metadata: {}
pro:
enabled: true
helmRelease:
chart:
version: 0.20.0-beta.15
values: |-
sync:
toHost:
ingresses:
enabled: true
# Checkout <https://vcluster.com/pro/docs/> for more config options
accessPoint:
ingress: {}
spaceTemplate:
metadata: {}
version: 0.0.0
access:
- verbs:
- '*'
subresources:
- '*'
users:
- admin
Kubernetes gateway API gRPCRoute​
apiVersion: management.loft.sh/v1
kind: App
metadata:
name: istio-base
spec:
displayName: Istio Base
owner:
team: loft-admins
recommendedApp:
- cluster
defaultNamespace: istio-system
icon: <https://seeklogo.com/images/I/istio-logo-92FF583709-seeklogo.com.png>
config:
chart:
name: base
version: 1.22.3
repoURL: <https://istio-release.storage.googleapis.com/charts>
---
apiVersion: management.loft.sh/v1
kind: App
metadata:
name: istiod
spec:
displayName: Istiod
owner:
team: loft-admins
recommendedApp:
- cluster
defaultNamespace: istio-system
icon: <https://seeklogo.com/images/I/istio-logo-92FF583709-seeklogo.com.png>
config:
chart:
name: istiod
version: 1.22.3
repoURL: <https://istio-release.storage.googleapis.com/charts>
---
apiVersion: management.loft.sh/v1
kind: App
metadata:
name: istio-ingress
spec:
displayName: Istio Ingress Gateway
owner:
team: loft-admins
recommendedApp:
- cluster
defaultNamespace: istio-ingress
icon: <https://seeklogo.com/images/I/istio-logo-92FF583709-seeklogo.com.png>
config:
chart:
name: gateway
version: 1.22.3
repoURL: <https://istio-release.storage.googleapis.com/charts>
---
apiVersion: management.loft.sh/v1
kind: VirtualClusterTemplate
metadata:
name: istio-virtual-service-example
spec:
displayName: Istio Virtual Service Example
description: >-
This virtual cluster template deploys a vCluster with generic sync configured to sync `virtualservices.networking.istio.io` from the vCluster to the host.
owner:
team: loft-admins
template:
metadata: {}
instanceTemplate:
metadata: {}
pro: {}
helmRelease:
chart:
version: 0.20.0-beta.15
values: |-
sync:
toHost:
ingresses:
enabled: true
# Checkout <https://vcluster.com/pro/docs/> for more config options
accessPoint:
ingress: {}
spaceTemplate:
metadata: {}
versions:
- template:
metadata:
annotations:
sleepmode.loft.sh/ignore-user-agents: argo*
instanceTemplate:
metadata:
labels:
env: '{{ .Values.env }}'
team: '{{ .Values.loft.project }}'
objects: |-
apiVersion: v1
kind: Namespace
metadata:
name: grpc-routing
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: grpc-routing
namespace: grpc-routing
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: grpc-routing-gateway
namespace: grpc-routing
labels:
example: grpc-routing
spec:
gatewayClassName: istio
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: grpc-routing
labels:
app: yages
example: grpc-routing
name: yages
spec:
selector:
matchLabels:
app: yages
replicas: 1
template:
metadata:
labels:
app: yages
spec:
serviceAccountName: grpc-routing
containers:
- name: grpcsrv
image: ghcr.io/projectcontour/yages:v0.1.0
ports:
- containerPort: 9000
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
namespace: grpc-routing
labels:
app: yages
example: grpc-routing
name: yages
spec:
ports:
- name: grpc
port: 9000
protocol: TCP
targetPort: 9000
selector:
app: yages
---
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
namespace: grpc-routing
name: yages
labels:
example: grpc-routing
spec:
parentRefs:
- name: grpc-routing-gateway
hostnames:
- "grpc-example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: yages
port: 9000
weight: 1
pro:
enabled: true
helmRelease:
chart:
version: 0.21.0-beta.5
values: |-
external:
platform:
autoSleep:
afterInactivity: 6000
autoDelete:
afterInactivity: 150000
sync:
toHost:
ingresses:
enabled: true
customResources:
gateways.gateway.networking.k8s.io:
enabled: true
grpcroutes.gateway.networking.k8s.io:
enabled: true
patches:
- path: .spec.parentRefs[*].name
reference:
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
- path: .spec.rules[*].backendRefs[*].name
reference:
kind: Service
apiVersion: v1
controlPlane:
statefulSet:
labels:
env: '{{ .Values.env }}'
team: '{{ .Values.loft.project }}'
# Use an embedded managed etcd server instead of using the default SQLite backend
backingStore:
etcd:
embedded:
enabled: true
distro:
k8s:
apiServer:
image:
tag: "{{ .Values.k8sVersion }}"
controllerManager:
image:
tag: "{{ .Values.k8sVersion }}"
coredns:
embedded: true
# Checkout <https://vcluster.com/pro/docs/> for more config options
accessPoint:
ingress: {}
spaceTemplate:
metadata: {}
parameters:
- variable: k8sVersion
label: k8sVersion
description: Please select Kubernetes version
options:
- v1.30.2
- v1.29.6
- v1.28.11
- v1.27.9
- v1.26.15
defaultValue: v1.29.6
- variable: env
label: Deployment Environment
description: >-
Environment for deployments for this vCluster used as cluster label
for Argo CD ApplicationSet Cluster Generator
options:
- dev
- qa
- prod
defaultValue: dev
version: 1.0.0
- template:
metadata: {}
instanceTemplate:
metadata: {}
pro:
enabled: true
helmRelease:
chart:
version: 0.20.0-beta.15
values: |-
sync:
toHost:
ingresses:
enabled: true
# Checkout <https://vcluster.com/pro/docs/> for more config options
accessPoint:
ingress: {}
spaceTemplate:
metadata: {}
version: 0.0.0
access:
- verbs:
- '*'
subresources:
- '*'
users:
- admin
Test the GRPCRoute configuration​
To verify your GRPCRoute is working correctly:
export GATEWAY_HOST=$(kubectl get gateway/grpc-routing-gateway -o jsonpath='{.status.addresses[0].value}')
grpcurl -plaintext -authority=grpc-example.com ${GATEWAY_HOST}:80 yages.Echo/Ping