Custom Resources

Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

vCluster allows you to sync custom resources from the virtual cluster to the host cluster. This allows you to sync arbitrary resources that are by default not synced by vCluster. This only works for resources that have a custom resource definition in the host cluster.

If those custom resources create other resources inside the host cluster, vCluster tries to find them and syncs them back to the host cluster as well. E.g. a cert-manager certificate creates a secret which syncs back automatically into the virtual cluster.

vCluster automatically adds the required cluster and namespace RBAC permissions for retrieving the custom resource definition and syncing the resources from the virtual cluster to the host cluster.

Only Namespace-Scoped Resource

This feature currently only works for namespace-scoped resources only.

Multi-Namespace-Mode

If you want to sync many custom resources, consider using multi-namespace-mode.

Enable custom resource syncing

To enable custom resource syncing from the virtual cluster to the host cluster, figure out what CRDs you want to sync via kubectl get crds. Add the name into the customResources section in the sync section. Even though vCluster syncs custom resources from the virtual cluster to the host cluster, the CRDs are also copied from the host cluster to the virtual cluster.

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true

Patches

You can modify the sync behavior with patches that target specific paths. Currently there are 2 different kinds of patches supported.

Wildcard patches

You can use * in paths to select all entries of an array or object, e.g. spec.containers[*].name or spec.containers[*].volumeMounts[*]. vCluster calls the patch multiple times.

Reference patches

A reference patch can be used to have a specific field of one resource point to a different resource that should get rewritten. vCluster automatically imports the referenced resource to the virtual cluster if it can find it in the host cluster. For example:

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true
        patches:
        - path: spec.secretName
          reference:
            apiVersion: v1
            kind: Secret

vCluster translates the path spec.secretName as it points to a secret. If the secret is created in the host cluster, vCluster automatically imports it into the virtual cluster.

Multi-Namespace-Mode

With multi-namespace-mode you only need to rewrite references that include a namespace. You can use the namespacePath option to specify the path of the namespace of the reference.

JavaScript expression patches

These are JavaScript ES6 compatible expression patches that can be used to change a field while syncing. You define how it changes when syncing from the virtual cluster into the host cluster or when syncing from the host cluster into the virtual cluster. To add a suffix to certificate DNS names you can:

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true
        patches:
        - path: spec.dnsNames[*]
          # specifies the sync direction here again, because you can also react on change for fromHost with an expression
          expression: '"www."+value'
          # optional reverseExpression, if omitted patches from host will be discarded for that path
          # reverseExpression: 'value.slice("my-prefix".length)'

There is also a variable called context besides value that can be used to access vCluster specific data:

• context.vcluster.name: Name of the virtual cluster
• context.vcluster.namespace: Namespace of the virtual cluster
• context.vcluster.config: Config of the virtual cluster, basically vcluster.yaml merged with the defaults
• context.hostObject: Host object (can be null if not available)
• context.virtualObject: Virtual object (can be null if not available)
• context.path: The matched path on the object, useful when using wildcard path selectors (*)

For example, to add www. to every DNS name specified in a cert-manager certificate in the path spec.dnsNames, you can use the following patch:

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true
        patches:
        - path: spec.dnsNames[*]
          # specifies the sync direction here again, because you can also react on change for fromHost
          expression: "value.startsWith('www.') ? value : `www.${value}`"
          # specifies how to sync back changes to the virtual cluster. If omitted will not sync back changes.
          reverseExpression: "value.startsWith('www.') ? value.slice(4) : value"

The patch creates a new certificate within the vCluster:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: name-within-vcluster
spec:
  dnsNames:
    - example.com

vCluster syncs the host cluster and applies your patch, creating this modified certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: synced-name
spec:
  dnsNames:
    - www.example.com # the patch added www. to this field

If you directly edit the certificate in the host cluster and change the domain:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: synced-name
spec:
  dnsNames:
    - www.other-domain.com # changed from www.example.com

vCluster detects the change, applies the reverse patch, and updates the certificate in your virtual cluster:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: name-within-vcluster
spec:
  dnsNames:
    - other-domain.com # the patch removed the www. from www.other-domain.com

Configure Kubernetes Gateway API sync

To use the Kubernetes Gateway API with custom resources, follow these steps:

Install Gateway CRD in the host

Install Gateway CRD

kubectl --context="${HOST_CTX}" get crd gateways.gateway.networking.k8s.io &> /dev/null || \
kubectl --context="${HOST_CTX}" apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Create waypoint gateway

Create a waypoint gateway configuration:

waypoint-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Apply it to your host cluster:

Create Waypoint Gateway

kubectl --context="${HOST_CTX}" create -f waypoint-gateway.yaml --namespace="${VCLUSTER_HOST_NAMESPACE}"

Once configured, you can configure your custom resources to sync Gateway API resources between the virtual and host clusters.

Config reference

customResources required {key: object} pro

CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work.

enabled required boolean pro

Enabled defines if this option should be enabled.

scope required string pro

Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported.

patches required object[] pro

Patches patch the resource according to the provided specification.

path required string pro

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression required string pro

Expression transforms the value according to the given JavaScript expression.

reverseExpression required string pro

ReverseExpression transforms the value according to the given JavaScript expression.

reference required object pro

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string pro

APIVersion is the apiVersion of the referenced object.

apiVersionPath required string pro

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string pro

Kind is the kind of the referenced object.

kindPath required string pro

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath required string pro

NamePath is the optional relative path to the reference name within the object.

namespacePath required string pro

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels required object pro

Labels treats the path value as a labels selector.