Synced Resources

This section lists all resources that can be synced or mirrored by vcluster currently in the table below. Those resources can be activated or deactivated via the values.yaml as described below, or with the --sync flag of the syncer. By default, certain resources are already activated and you can either disable the default resources or tell vcluster to sync other supported resources as well.

Resource	Description	Default Enabled
services	Mirrors services between host and virtual cluster	Yes
endpoints	Mirrors endpoints between host and virtual cluster	Yes
configmaps	Mirrors used configmaps by pods between host and virtual cluster	Yes
secrets	Mirrors used secrets by ingresses or pods between host and virtual cluster	Yes
events	Syncs events from host cluster to virtual cluster	Yes
pods	Mirrors pods between host and virtual cluster	Yes
persistentvolumeclaims	Mirrors persistent volume claims between host and virtual cluster	Yes
fake-nodes	Creates fake nodes based on spec.nodeName fields of synced pods. Requires no cluster role	Yes
fake-persistentvolumes	Creates fake persistent volumes based on spec.volumeName of persistent volume claims. Requires no cluster role	Yes
ingresses	Mirrors ingresses between host and virtual cluster. Automatically tries to detect the supported ingress version (networking.k8s.io/v1 or networking.k8s.io/v1beta1)	No
ingressclasses	Syncs IngressClasses from host cluster to virtual cluster. This is automatically enabled when Ingresses sync is enabled.	No *
nodes	Syncs real nodes from host cluster to virtual cluster. If enabled, implies that fake-nodes is disabled. For more information see nodes.	No
persistentvolumes	Mirrors persistent volumes from vcluster to host cluster and dynamically created persistent volumes from host cluster to virtual cluster. If enabled, implies that fake-persistentvolumes is disabled. For more information see storage.	No
storageclasses	Syncs created storage classes from virtual cluster to host cluster	No
hoststorageclasses	Syncs real storage classes from host cluster to virtual cluster. This is only needed if you require to be able to get/list StorageClasses from vcluster API server. Host storage classes can be used in PersistentVolumes and PersistentVolumeClaims without syncing them to the virtual cluster. This option was formerly named "legacy-storageclasses".	No
priorityclasses	Syncs created priority classes from virtual cluster to host cluster	No
networkpolicies	Syncs created network policies from virtual cluster to host cluster	No
volumesnapshots	Enables volumesnapshot, volumesnapshotcontents and volumesnapshotclasses support. Syncing behaves similar to persistentvolumeclaims, persistentvolumes and storage classes. For more information see storage.	No
poddisruptionbudgets	Syncs created poddisruptionbudgets from virtual cluster to host cluster	No
serviceaccounts	Syncs created service accounts from virtual cluster to host cluster. This is useful for using IAM roles for service accounts with vcluster	No
csidrivers	Mirrors CSIDriver objects from host cluster to vcluster. Enabled automatically when virtual scheduler is enabled. Disabling this syncer while using virtual scheduler may result in incorrect pod scheduling.	No *
csinodes	Mirrors CSINode objects from host cluster to vcluster. Enabled automatically when virtual scheduler is enabled. Disabling this syncer while using virtual scheduler may result in incorrect pod scheduling.	No *
csistoragecapacities	Mirrors CSIStorageCapacity Objects from host cluster to vcluster if the .nodeTopology matches a synced node. Enabled automatically when virtual scheduler is enabled. Disabling this syncer while using virtual scheduler may result in incorrect pod scheduling.	No *

* refer to the description column for claryfying information about default behavior.

By default, vcluster runs with a minimal set of RBAC permissions to allow execution in restricted environments. Certain resources require extra permissions, which will be automatically given to the vcluster ServiceAccount if you enable the resource sync with the associated helm value.

Enable or disable synced resources#

To enable a resource syncronization, for example persistent volumes, and automatically create the necessary RBAC permissions, add the following to your values.yaml:

sync:
  persistentvolumes:
    enabled: true

then create or upgrade the vcluster with:

vcluster create my-vcluster --upgrade -f values.yaml

To disable a resource that is synced by default, for example if you don't want to sync services, set the following in your values.yaml:

sync:
  services:
    enabled: false

then create or upgrade the vcluster with:

vcluster create my-vcluster --upgrade -f values.yaml

Correct Cluster Functionality

Disabling certain resources such as services, endpoints or pods can lead to a non-functional virtual Kubernetes cluster, so be careful with what resources you are deactivating.

Sync all Secrets and Configmaps#

With the new generic sync, vcluster currently only knows about a couple of resources that actually use secrets / configmaps and will try to sync only those into the host cluster, but this allows syncing of all secrets and configmaps to avoid the problem that needed secrets / configmaps are not synced to the host cluster. To enable this, simply add the following values to the helm chart / vcluster cli options:

sync:
  secrets:
    all: true
  configmaps:
    all: true

Extra Pod Options#

By default ephemeral containers and readiness gates will not be synced by vcluster, as they require additional permissions. To enable those, please activate those within your values.yaml:

sync:
  pods:
    enabled: true
    # Sync ephemeralContainers to host cluster
    ephemeralContainers: true
    # Sync readiness gates to host cluster
    status: true

Sync other resources#

Syncing other resources such as deployments, statefulsets and namespaces is usually not needed as those just control lower level resources and since those lower level resources are synced the cluster can function correctly.

However, there might be cases though where custom syncing of resources might be needed or beneficial. In order to accomplish this, vcluster provides an SDK to develop your own resource syncers as plugins. To find out more, please take a look at the plugins documentation.

Generic sync#

Besides the plugins, vcluster provides a way to define additional resources that should be synced in a generic and declarative way with just a few lines of a YAML configuration. This feature is a successor to the vcluster-generic-crd-sync-plugin project and is included since v0.14.0 release. The full range of the generic sync features is available only in the vcluster created in the "multi-namespace mode", see the "Multi-namespace mode" chapter for details.

You will need to declare which CRD Kinds you would like to sync from the virtual cluster to the host cluster, or vice versa, and the vcluster will automatically copy the CRD definition from the host cluster into vcluster at the start. Then it will take care of watching the resources of the predefined Kinds and execute the synchronization logic based on the configuration provided to it. The vcluster may automatically transform the resource metadata(such as name, namespace, labels, etc.) as is common for resources synced by vcluster. In addition to the implicit metadata transformations, you can configure transformations that will be performed on other fields of the resource, and these will depend on the meaning of those fields. You may also declare which fields will be copied in the opposite direction, from the synced resource to the original one. Many controllers create Kubernetes resources as a result of custom resources, for example, cert-manager creates Secrets based on Certificate custom resources, and this feature will allow you to sync these resources from the host cluster into the virtual one. The following chapters describe the configuration options in more detail.

tip

You may find configuration examples in the "generic-sync-examples" folder in the vcluster repo.

Alpha feature

Generic sync feature is currently in an alpha state. This is an advanced feature that requires more permissions in the host cluster, and as a result, it can potentially cause significant disruption in the host cluster.

Configuration syntax#

The helm values snippet below shows an example of the generic sync configuration and related RBAC roles. There you can notice some key fields nested under .sync.generic value:

the RBAC namespaced role and cluster scoped clusterRole required for the plugin - these would be adjusted to fit the needs of your use case and the configuration that you define. Note that when the "Multi-namespace mode" is used, the namespaced role will become ClusterRole.
the config field, which will populate the CONFIG environment variable of the vcluster syncer container - this must be a string with valid YAML formatting. It uses a custom syntax to define the behavior of the plugin.

sync:
  generic:
    clusterRole:
      extraRules:
        - apiGroups: ["apiextensions.k8s.io"]
          resources: ["customresourcedefinitions"]
          verbs: ["get", "list", "watch"]
    role:
      extraRules:
        # Example for Cert Manager
        - apiGroups: ["cert-manager.io"]
          resources: ["issuers", "certificates", "certificaterequests"]
          verbs: ["create", "delete", "patch", "update", "get", "list", "watch"]
    config: |-
      version: v1beta1
      export:
      # ...

Virtual to Host sync#

We use the top-level export field in the configuration to declare which virtual resources we want to sync to the host cluster. Each item in the export array defines the resource via apiVersion and kind strings. Each apiVersion and kind pair can have only one entry in the export array. The patches field allows you to define how are certain fields of the synced resource modified before its creation(or update) in the host cluster.
The reversePatches field allows you to declare how changes to certain fields(implicitly this is done for the status) of the synced resource(the one created in the host cluster) are propagated back to the original resource in the virtual cluster. Besides the status, only the fields referenced in the copyFromObject reverse patch operations are propagated. Both these fields follow the same syntax, as documented in the "Patch syntax" chapter of this doc.

Example:

sync:
  generic:
    config: |-
      version: v1beta1
      export:
        - apiVersion: cert-manager.io/v1
          kind: Certificate
          patches:
            - op: rewriteName
              path: spec.issuerRef.name
            - op: rewriteName
              path: spec.secretName
          reversePatches:
            # Implicit reverse patch for status would be declared like so:
            # - op: copyFromObject
            #   fromPath: status
            #   path: status

info

Only the namespaced resources are supported at this time.

Selector for a generic Virtual to Host sync
You can limit which resources will be synced from the virtual cluster by configuring the selector array. The virtual resource is synced when it matches one or more selectors, or when the selector field is empty. Supported selector types are:
labelSelector - the key: value map of the resource labels. All of the defined label key and values should match on the resource in the virtual cluster to be synced. Example:

sync:
  generic:
    config: |-
      version: v1beta1
      export:
        - apiVersion: cert-manager.io/v1
          kind: Certificate
          selector: 
            - labelSelector: 
              - "label-key": "label-value"

Host to Virtual sync#

We use the top-level import field in the configuration to declare which host resources we want to sync to the virtual cluster. Each item in the import array defines the resource via apiVersion and kind strings. Each apiVersion and kind pair can have only one entry in the import array. The patches field allows you to define how are certain fields of the synced resource modified before its creation(or update) in the virtual cluster.
The reversePatches field allows you to declare how changes to certain fields of the synced resource(in this case, the one created in the virtual cluster) are propagated back to the original resource in the host cluster. Only the fields referenced in the copyFromObject reverse patch operations are propagated. Both these fields follow the same syntax, as documented in the "Patch syntax" chapter of this doc.

Example:

sync:
  generic:
    config: |-
      version: v1beta1
      import:
        - kind: Secret
          apiVersion: v1
        - kind: IngressClass
          apiVersion: networking.k8s.io/v1

info

The sync from Host to Virtual cluster is supported only in "Multi-namespace mode"

Patch syntax#

The patch defines how will the vcluster behave when syncing each resource to and from the host cluster. Generally, a patch is defined by the field path and op(operation) that should be performed on said field.
An array of conditions may also be set, and in such case, the field value will be modified by a patch only if the field value matches all the conditions.
Some operation types may utilize additional fields, and these will be explained in the next chapter.

Patch operations

op	Support	Description
copyFromObject	all	Copy value of the field referenced in the `fromPath` from the originating object to the `path` field of the destination object. The `fromPath` can be omitted, in such case, it will default to the same field path as referenced in the `path`.
add	all	Add contents of the `value` into the `path` field. The `value` can be either scalar or a complex object.
replace	all	Replace the contents of the `path` field with the contents of the `value`. The `value` can be either scalar or a complex object.
remove	all	Remove the contents of the `path` field
rewriteName	V->H	Replaces the contents of the `path` field with transformed content based on the namespace of the synced resource. This is typically done on the fields that refer to a resource name, and on the `.metadata.name` as well(implicit). This is done to avoid naming collisions when syncing resources to the host cluster, but it is not necessary when using the "Multi-namespace mode". As an example, the "logstash" value of a resource in the "logging" namespace of the vcluster named "vc" is rewritten to "logstash-x-logging-x-vc". If the resulting length of the value would be over 63 characters, the last 10 characters will be replaced with a hash of the full value.
rewriteName + namePath + namespacePath	V->H	Similar to `rewriteName`, but with an addition of the `namePath` and/or `namespacePath`. This is used when a field of the synced resource is referencing a different resource via namespace and name via two separate fields. When using this option you would set the `path` to reference a field that is a common parent of both `namePath` and `namespacePath`, and these two fields would then contain just the relative path. For example, `path: spec.includes` + `namePath: name` + `namespacePath: namespace` for a resource that contains name in `spec.includes.name` and namespace in `spec.includes.namespace`.
rewriteName + regex	V->H	Similar to `rewriteName`, but with an addition of the `regex` option for the patch. This is used when a string contains not just the resource name, but optionally a namespace, and other characters. For example, a string containing "namespace/name" can be correctly rewritten with the addition of this configuration option - `regex: "$NAMESPACE/$NAME"`. The vcluster uses Go regular expressions to recognize the name part with the "NAME" capture group (can be written as `$NAME`), and the namespace with the "NAMESPACE" capture group (can be written as `$NAMESPACE`).
rewriteLabelKey	V->H	The keys of the `.metadata.labels` of the synced resources are rewritten by vcluster and plugins. This patch type allows you to rewrite the key references in the same way, so the fields that are referencing labels will still reference correct labels in their rewritten form. For example, the label key-value pair "app: curl" is rewritten to "vcluster.loft.sh/label-vcluster-x-a172cedcae: curl", so with this patch operation you can rewrite a field that contains "app" to "vcluster.loft.sh/label-vcluster-x-a172cedcae, and the controllers operating on the synced resources will work with this label just as expected. This is not necessary when using the ["Multi-namespace mode"].(#multi-namespace-mode).
rewriteLabelSelector	V->H	This operation exists for the same reasons as described for the rewriteLabelKey operation. It is intended to be used for the key-value map fields that represent a label selector. This patch operation will rewrite all keys in the field referenced by `path` to the expected format for the label keys, and it will also add additional key-value pairs(with virtual namespace and vcluster name) to avoid naming conflicts. This is not necessary when using the ["Multi-namespace mode"].
rewriteLabelExpressionsSelector	V->H	Similar to the `rewriteLabelSelector`, but expects `path` reference a field with the `matchLabels` and `matchExpressions` sub-fields, which will have the label keys rewritten just as described for `rewriteLabelKey`. This is not necessary when using the ["Multi-namespace mode"].

V->H - patch operation is supported only for patches, or reverse patches, that are executed in the virtual to host direction.

Example of various patch operations
The example below is a useful configuration for pushing service metrics into a Prometheus instance installed in the host cluster.
You can refer to ServiceMonitor spec documentation to understand the example better.

sync:
  generic:
    config: |-
      version: v1beta1
      export:
        - apiVersion: monitoring.coreos.com/v1
          kind: ServiceMonitor
          patches:
            - op: add
              path: .metadata.labels
              value:
                prometheus-instance: default-instance
            - op: copyFromObject
              fromPath: .metadata.labels['prometheus-instance'] # could be omitted when equal to path
              path: .metadata.labels['prometheus-instance']
            - op: replace
              path: .spec.namespaceSelector
              value:
                any: false
                matchNames: []
            - op: rewriteName
              path: .spec.endpoints[*]
            - op: rewriteLabelKey
              path: .spec.jobLabel
            - op: rewriteLabelKey
              path: .spec.targetLabels[*]
            - op: rewriteLabelKey
              path: .spec.podTargetLabels[*]
            - op: rewriteLabelExpressionsSelector
              path: .spec.selector
            # A regex example, e.g. if we have a field in "namespace/name" string
            - op: rewriteName
              path: .metadata.annotations['custom-one-namespace-slash-name']
              regex: >
                ^$NAMESPACE/$NAME$

Patch conditions
A patch can be applied conditionally by populating the conditions field of the patch definition. The condition will be checked either on a field referenced relative to the patch path via the subPath field of the condition or referenced as an absolute path in the synced resource via the path field of the condition.
A condition can have one of these three fields set to check the value respectively:

equal - either a scalar value or a complex object can be used
notEqual - either a scalar value or a complex object can be used
empty - a boolean (true or false) value Examples:

...
patches:
    - op: add
        path: .metadata.labels
        value:
            prometheus-instance: default-instance
        conditions:
            - path: .metadata.labels['prometheus-instance']
                equal: "forbidden-instance"
    - op: copyFromObject
        fromPath: .metadata.labels['prometheus-instance'] # could be omitted when equal to path
        path: .metadata.labels['prometheus-instance']
        conditions:
            - subPath: "."
              notEqual: "forbidden-instance"
    - op: add
        path: .metadata.labels
        value:
            mandatory: setIt
        conditions:
            - path: .metadata.labels['mandatory']
                empty: true

More Examples#

A list of sample configurations can be found here - vcluster generic-sync-examples

Multi-namespace mode#

In this mode vcluster diverges from the architecture described previously. By default, all namespaced resources that need to be synced to the host cluster are created in the namespace where vcluster is installed. But in multi-namespace mode vcluster will create a namespace in the host cluster for each namespace in the virtual cluster. The namespace name is modified to avoid conflicts between multiple vcluster instances in the same host, but the synced namespaced resources are created with the same name as in the virtual cluster. To enable this mode use the following helm value:

multiNamespaceMode:
  enabled: true

This mode must be enabled during vcluster creation.

Enabling, or disabling, it on an existing vcluster instance will force it into an inconsistent state.

Alpha feature

Multi-namespace mode is currently in an alpha state. This is an advanced feature that requires more permissions in the host cluster, and as a result, it can potentially cause significant disruption in the host cluster.