Limited vCluster Tenancy Configuration Support

This feature is only available when using the following worker node types:

Host Nodes

Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Sync custom resources from host to virtual cluster

vCluster allows you to sync custom resources from the host cluster to the virtual cluster. This feature creates read-only copies of custom resources inside your virtual cluster, making them available to applications and users without giving them direct access to the host cluster.

When you enable custom resource syncing, vCluster first copies the CustomResourceDefinition (CRD) from the host to the virtual cluster, then begins syncing the actual custom resource instances. This is particularly useful for sharing cluster-wide resources like cluster-scoped custom resources or ClusterStores from external-secrets with your virtual cluster workloads.

The synced resources are read-only in the virtual cluster. If you modify them in the host cluster, vCluster automatically updates the virtual cluster copies. However, changes made to these resources within the virtual cluster are not persisted.

If you need to sync resources from the virtual cluster to the host cluster instead, see syncing custom resources to the host cluster.

No need to configure RBAC

vCluster automatically adds the required cluster RBAC permissions for retrieving the CustomResourceDefinition and syncing the resources from the host to the virtual cluster.

Cluster-scoped example

Cluster-scoped custom resources exist at the cluster level and are not tied to any specific namespace. These are the simplest to sync because they don't require namespace mapping.

To sync cluster-scoped custom resources from the host cluster, configure the resource in your vcluster.yaml file:

configure cluster-scoped CRD sync from host

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Cluster

By default, vCluster does not require you to specify an API version when syncing custom resources. When a CRD has multiple versions and you don't specify a version, vCluster uses the storage version.

If you want to sync a specific version of a CRD that is not the storage version, you can explicitly specify it in the sync configuration.

Versioned example

To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a /, using the format <resource>/<version> in your configuration.

The specified version must exist in the host cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the host cluster's storage version, vCluster automatically converts custom resources between the two versions.

For example, to use version v1 of the certificaterequests.cert-manager.io CRD, specify:

Configure cluster-scoped CR sync from host

sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io/v1:
        enabled: true
        scope: Cluster

Unique CRD versions

vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the virtual cluster fails to start.

Upgrade custom resources with explicit CRD API versions

vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the virtual cluster fails to start.

vCluster modifies the CRD inside the virtual cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the virtual cluster or change the sync configuration.

Upgrade from no version to specified version

When upgrading from an unversioned CRD to a versioned one (certificaterequests.cert-manager.io → certificaterequests.cert-manager.io/v1), vCluster checks the CRD's storage version. If the storage version is not v1, vCluster updates the CRD to include v1 as a new version in addition to the current storage version.

Upgrade from one specified version to another specified version

When upgrading between versioned CRDs (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io/v2), the CRD is updated to include the new version v2, and the old version v1 is kept as well.

Upgrade from a specified version to no version specified

When upgrading from a versioned CRD to an unversioned one (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io), the behavior depends on the host's storage version:

• If the CRD's storage version in the host is v1, nothing happens and the CRD in the virtual cluster remains as certificaterequests.cert-manager.io/v1.
• If the storage version is not v1, vCluster updates the CRD to include the storage version as a new version in addition to v1.

Namespace-scoped custom resource definitions

By default, namespace-scoped custom resource syncing is disabled.

Enabling this feature allows you to sync namespaced custom resources from specific namespaces in the host cluster to corresponding namespaces in the virtual cluster.

Configure namespace-scoped CRD sync from host

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CustomObjects from "foo" namespace
            # to the "bar" namespace in a virtual cluster. CustomObjects names are unchanged.
            "foo/*": "bar/*"

Key behaviors for namespace scoped syncing:

• You can modify the name of the synced resource in the virtual cluster.
• You cannot sync from all namespaces in the host cluster.
• Sync is one-directional, from host to virtual. If you modify an object in the host, vCluster syncs the change to the virtual object.
• When you delete a virtual object, vCluster re-creates it if the host object still exists.
• When you delete a host object, vCluster deletes the virtual object.

vCluster creates namespaces in the virtual cluster automatically during the sync if they don't already exist.

Prerequisites

Ensure you have the following:

• All specified namespaces must exist in the host cluster when vCluster starts.
• The CustomResourceDefinition must exist in the host cluster when vCluster starts.

Example

No need to configure RBAC

vCluster automatically adds the required cluster RBAC permissions for retrieving the CustomResourceDefinition and syncing the resources from the host to the virtual cluster.

For namespace scoped custom resources, you must specify mappings.byName in the config. This tells vCluster which host resources should be synced and where to place them in the virtual cluster.

vCluster creates namespaces in the virtual cluster automatically during the sync if they don't already exist.

vCluster cannot sync custom resources that were already synced from virtual to host. It skips these resources.

The following is an example with a generic custom resource.

To sync all custom resources from a given namespace in the host to a given namespace in the virtual cluster, use the "namespace/*" wildcard:

Configure custom resource sync from host namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CustomObjects from "foo" namespace
            # to the "bar" namespace in a virtual cluster. CustomObjects names are unchanged.
            "foo/*": "bar/*"

To sync only specific custom resources from namespaces, provide namespace/name as the key and value:

Configure custom resource sync from host for one object

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs CustomObject named "my-object" from "foo" host namespace
            # to the "bar" namespace in virtual.
            "foo/my-object": "bar/my-object"

There is also a syntax to sync all custom resources from the virtual cluster's own host namespace to the virtual namespace. Since the virtual cluster's namespace is not always known upfront (for example, when vCluster is created by the platform), "" (empty string) is treated as "vCluster's own host namespace":

configure custom resource sync from host for virtual cluster's namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CustomObjects from virtual cluster's host namespace
            # to "my-virtual" namespace in a virtual cluster.
            "": "my-virtual"

You can also specify only a few custom resources from the virtual cluster's own host namespace:

configure custom resource sync from host for objects in virtual cluster's namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs CustomObject named "my-object" from virtual cluster's host namespace
            # to "my-virtual-namespace" in a virtual cluster.
            "/my-object": "my-virtual/my-object"

It's also possible to modify the custom resource name during the sync:

configure custom resource sync from host and modify name and namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs "foo" CustomObject from "system" namespace in the host
            # as "my-foo" in "my-virtual" namespace in a virtual cluster.
            "system/foo": "my-virtual/my-foo"

Patches

You can use patches to transform data as it syncs from the host to the virtual cluster.

When syncing custom resources from host to virtual, you can use reverseExpression to modify the data during the sync process. The host cluster remains unchanged, but the virtual cluster gets the modified data.

note

expressions (used for virtual-to-host syncing) have no effect when syncing from host to virtual.

The following vcluster.yaml shows how patches are used:

configure custom resource sync from host with patches

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            "default/my-object": "barfoo2/object-my"
        patches:
          - path: metadata.annotations[*]
            # optional reverseExpression to reverse the change from the host cluster
            reverseExpression: "value.startsWith('www.') ? value.slice(4) : value"

In the example:

• The custom resource in the host namespace default named my-object is synced to the namespace barfoo2 in virtual and named object-my.

• If the default/my-object host object has an annotation with a value that starts with www. (for example: my-address: www.loft.sh), then the synced object in the virtual cluster barfoo2/object-my has the annotation my-address: loft.sh.

From host namespaced CustomResource example

This guide shows how to sync k8s namespaced CustomResources from host cluster. Example CRD is used in this guide.

Set up cluster contexts

Setting up the host and virtual cluster contexts makes it easier to switch between them.

set up kubectl contexts

export HOST_CTX="your-host-context"
export VCLUSTER_CTX="vcluster-ctx"

then, create a namespace in your host cluster, use foobar2 as an example:

create namespace

kubectl --context="${HOST_CTX}" create namespace foobar2

tip

You can find your contexts by running kubectl config get-contexts

Create CustomResourceDefinition in the host

Saved following Custom Resource Definition:

example-crd.yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: examples.demo.loft.sh
spec:
  group: demo.loft.sh
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer
      additionalPrinterColumns:
      - name: Image
        type: string
        description: The image of an example
        jsonPath: .spec.image
      - name: Replicas
        type: integer
        description: The number of replicas in example
        jsonPath: .spec.replicas
  scope: Namespaced
  names:
    plural: examples
    singular: example
    kind: Example

save this file locally and then apply it in the host cluster:

Create Example CRD in the host

kubectl --context="${HOST_CTX}" create -f example-crd.yaml

Enable from host syncing for your CustomResource

Enable the from host syncing for Example CustomResources in your virtual cluster configuration:

Enable from host syncing for CustomResource

sync:
  fromHost:
    customResources:
      examples.demo.loft.sh:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            "": "default"

This configuration:

• Enables from host syncing of the examples.demo.loft.sh from the vCluster's host namespace
• Automatically configures RBAC permissions for vCluster, so it can get, watch and list Examples in vCluster's host namespace.
• Syncs all Examples from vCluster host namespace to the default namespace in your vCluster.

create virtual cluster

Create or update a virtual Cluster following the vCluster quick start guide.

Sync namespaced CustomResource to virtual cluster

1. First, you create an example that you want to sync in the host cluster:

   Copy this file and save it locally as example-cr.yaml

   apiVersion: demo.loft.sh/v1
   kind: Example
   metadata:
     name: my-example
     namespace: vcluster
   spec:
     image: "my-image:latest"
     replicas: 2

   then, create an example in the host cluster:

   Create example in the host

   kubectl --context="${HOST_CTX}" create -f example-cr.yaml

Ensure that CustomResource got synced to the virtual cluster

Your CustomResource should be now accessible in the virtual cluster. Keep in mind, that any edit made in the virtual object is overwritten by the host object data.

2. Check CustomResource in the virtual cluster:

   Get synced CustomResource

   kubectl --context="${VCLUSTER_CTX}" get examples.demo.loft.sh --namespace default

   you should see similar output:

   examples in vCluster

   NAME         IMAGE             REPLICAS
   my-example   my-image:latest   2

Edit CustomResource in the host

3. Now, you can edit your example in the host and see that the change is synced to the vCluster. To set replicas to 4, run:

   Patch Example CR

   kubectl --context="${HOST_CTX}" patch examples.demo.loft.sh my-example --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 4}]'  --namespace vcluster

Verify that Example is updated in vCluster

4. Virtual Cluster Check number of replicas

   Check example

   kubectl --context="${VCLUSTER_CTX}" get --namespace default examples.demo.loft.sh

   you should see number of replicas updated from host object:

   Example updated in vCluster

   NAME         IMAGE             REPLICAS
   my-example   my-image:latest   4

Config reference

customResources required {key: object}

CustomResources defines what custom resources should get synced read-only to the virtual cluster from the host cluster. vCluster will automatically add any required RBAC to the vCluster cluster role.

enabled required boolean

Enabled defines if this option should be enabled.

scope required string

Scope defines the scope of the resource

patches required object[]

Patches patch the resource according to the provided specification.

path required string

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression required string

Expression transforms the value according to the given JavaScript expression.

reverseExpression required string

ReverseExpression transforms the value according to the given JavaScript expression.

reference required object

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string

APIVersion is the apiVersion of the referenced object.

apiVersionPath required string

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string

Kind is the kind of the referenced object.

kindPath required string

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath required string

NamePath is the optional relative path to the reference name within the object.

namespacePath required string

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels required object

Labels treats the path value as a labels selector.

mappings required object

Mappings for Namespace and Object

byName required object

ByName is a map of host-object-namespace/host-object-name: virtual-object-namespace/virtual-object-name. There are several wildcards supported:

1. To match all objects in host namespace and sync them to different namespace in vCluster: byName: "foo/": "foo-in-virtual/"
2. To match specific object in the host namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object"
3. To match specific object in the host namespace and sync it to the same namespace with different name: byName: "foo/my-object": "foo/my-virtual-object"
4. To match all objects in the vCluster host namespace and sync them to a different namespace in vCluster: byName: "": "my-virtual-namespace/*"
5. To match specific objects in the vCluster host namespace and sync them to a different namespace in vCluster: byName: "/my-object": "my-virtual-namespace/my-object"