Synced Resources
This section lists all resources that can be synced or mirrored by vcluster currently in the table below. Those resources can be activated or deactivated via the values.yaml
as described below, or with the --sync
flag of the syncer. By default, certain resources are already activated and you can either disable the default resources or tell vcluster to sync other supported resources as well.
Resource | Description | Default Enabled |
---|---|---|
services | Mirrors services between host and virtual cluster | Yes |
endpoints | Mirrors endpoints between host and virtual cluster | Yes |
configmaps | Mirrors used configmaps by pods between host and virtual cluster | Yes |
secrets | Mirrors used secrets by ingresses or pods between host and virtual cluster | Yes |
events | Syncs events from host cluster to virtual cluster | Yes |
pods | Mirrors pods between host and virtual cluster | Yes |
persistentvolumeclaims | Mirrors persistent volume claims between host and virtual cluster | Yes |
fake-nodes | Creates fake nodes based on spec.nodeName fields of synced pods. Requires no cluster role | Yes |
fake-persistentvolumes | Creates fake persistent volumes based on spec.volumeName of persistent volume claims. Requires no cluster role | Yes |
ingresses | Mirrors ingresses between host and virtual cluster. Automatically tries to detect the supported ingress version (networking.k8s.io/v1 or networking.k8s.io/v1beta1) | No |
ingressclasses | Syncs IngressClasses from host cluster to virtual cluster. This is automatically enabled when Ingresses sync is enabled. | No * |
nodes | Syncs real nodes from host cluster to virtual cluster. If enabled, implies that fake-nodes is disabled. For more information see nodes. | No |
persistentvolumes | Mirrors persistent volumes from vcluster to host cluster and dynamically created persistent volumes from host cluster to virtual cluster. If enabled, implies that fake-persistentvolumes is disabled. For more information see storage. | No |
storageclasses | Syncs created storage classes from virtual cluster to host cluster | No |
hoststorageclasses | Syncs real storage classes from host cluster to virtual cluster. This is only needed if you require to be able to get/list StorageClasses from vcluster API server. Host storage classes can be used in PersistentVolumes and PersistentVolumeClaims without syncing them to the virtual cluster. This option was formerly named "legacy-storageclasses". | No |
priorityclasses | Syncs created priority classes from virtual cluster to host cluster | No |
networkpolicies | Syncs created network policies from virtual cluster to host cluster | No |
volumesnapshots | Enables volumesnapshot, volumesnapshotcontents and volumesnapshotclasses support. Syncing behaves similar to persistentvolumeclaims, persistentvolumes and storage classes. For more information see storage. | No |
poddisruptionbudgets | Syncs created poddisruptionbudgets from virtual cluster to host cluster | No |
serviceaccounts | Syncs created service accounts from virtual cluster to host cluster. This is useful for using IAM roles for service accounts with vcluster | No |
csidrivers | Mirrors CSIDriver objects from host cluster to vcluster. Enabled automatically when virtual scheduler is enabled. Disabling this syncer while using virtual scheduler may result in incorrect pod scheduling. | No * |
csinodes | Mirrors CSINode objects from host cluster to vcluster. Enabled automatically when virtual scheduler is enabled. Disabling this syncer while using virtual scheduler may result in incorrect pod scheduling. | No * |
csistoragecapacities | Mirrors CSIStorageCapacity Objects from host cluster to vcluster if the .nodeTopology matches a synced node. Enabled automatically when virtual scheduler is enabled. Disabling this syncer while using virtual scheduler may result in incorrect pod scheduling. | No * |
* refer to the description column for claryfying information about default behavior.
By default, vcluster runs with a minimal set of RBAC permissions to allow execution in restricted environments. Certain resources require extra permissions, which will be automatically given to the vcluster ServiceAccount if you enable the resource sync with the associated helm value.
#
Enable or disable synced resourcesTo enable a resource syncronization, for example persistent volumes, and automatically create the necessary RBAC permissions, add the following to your values.yaml
:
then create or upgrade the vcluster with:
To disable a resource that is synced by default, for example if you don't want to sync services, set the following in your values.yaml
:
then create or upgrade the vcluster with:
Correct Cluster Functionality
Disabling certain resources such as services, endpoints or pods can lead to a non-functional virtual Kubernetes cluster, so be careful with what resources you are deactivating.
#
Sync all Secrets and ConfigmapsWith the new generic sync, vcluster currently only knows about a couple of resources that actually use secrets / configmaps and will try to sync only those into the host cluster, but this allows syncing of all secrets and configmaps to avoid the problem that needed secrets / configmaps are not synced to the host cluster. To enable this, simply add the following values to the helm chart / vcluster cli options:
#
Extra Pod OptionsBy default ephemeral containers and readiness gates will not be synced by vcluster, as they require additional permissions. To enable those, please activate those within your values.yaml:
#
Sync other resourcesSyncing other resources such as deployments, statefulsets and namespaces is usually not needed as those just control lower level resources and since those lower level resources are synced the cluster can function correctly.
However, there might be cases though where custom syncing of resources might be needed or beneficial. In order to accomplish this, vcluster provides an SDK to develop your own resource syncers as plugins. To find out more, please take a look at the plugins documentation.
#
Generic syncBesides the plugins, vcluster provides a way to define additional resources that should be synced in a generic and declarative way with just a few lines of a YAML configuration. This feature is a successor to the vcluster-generic-crd-sync-plugin project and is included since v0.14.0 release. The full range of the generic sync features is available only in the vcluster created in the "multi-namespace mode", see the "Multi-namespace mode" chapter for details.
You will need to declare which CRD Kinds you would like to sync from the virtual cluster to the host cluster, or vice versa, and the vcluster will automatically copy the CRD definition from the host cluster into vcluster at the start. Then it will take care of watching the resources of the predefined Kinds and execute the synchronization logic based on the configuration provided to it. The vcluster may automatically transform the resource metadata(such as name, namespace, labels, etc.) as is common for resources synced by vcluster. In addition to the implicit metadata transformations, you can configure transformations that will be performed on other fields of the resource, and these will depend on the meaning of those fields. You may also declare which fields will be copied in the opposite direction, from the synced resource to the original one. Many controllers create Kubernetes resources as a result of custom resources, for example, cert-manager creates Secrets based on Certificate custom resources, and this feature will allow you to sync these resources from the host cluster into the virtual one. The following chapters describe the configuration options in more detail.
tip
You may find configuration examples in the "generic-sync-examples" folder in the vcluster repo.
Alpha feature
Generic sync feature is currently in an alpha state. This is an advanced feature that requires more permissions in the host cluster, and as a result, it can potentially cause significant disruption in the host cluster.
#
Configuration syntaxThe helm values snippet below shows an example of the generic sync configuration and related RBAC roles. There you can notice some key fields nested under .sync.generic
value:
- the RBAC namespaced
role
and cluster scopedclusterRole
required for the plugin - these would be adjusted to fit the needs of your use case and the configuration that you define. Note that when the "Multi-namespace mode" is used, the namespaced role will become ClusterRole. - the
config
field, which will populate theCONFIG
environment variable of the vcluster syncer container - this must be a string with valid YAML formatting. It uses a custom syntax to define the behavior of the plugin.
#
Virtual to Host syncWe use the top-level export
field in the configuration to declare which virtual resources we want to sync to the host cluster. Each item in the export
array defines the resource via apiVersion
and kind
strings. Each apiVersion
and kind
pair can have only one entry in the export
array. The patches
field allows you to define how are certain fields of the synced resource modified before its creation(or update) in the host cluster.
The reversePatches
field allows you to declare how changes to certain fields(implicitly this is done for the status
) of the synced resource(the one created in the host cluster) are propagated back to the original resource in the virtual cluster. Besides the status, only the fields referenced in the copyFromObject
reverse patch operations are propagated.
Both these fields follow the same syntax, as documented in the "Patch syntax" chapter of this doc.
Example:
info
Only the namespaced resources are supported at this time.
Selector for a generic Virtual to Host sync
You can limit which resources will be synced from the virtual cluster by configuring the selector
array. The virtual resource is synced when it matches one or more selectors, or when the selector
field is empty. Supported selector types are:
labelSelector
- the key: value
map of the resource labels. All of the defined label key and values should match on the resource in the virtual cluster to be synced. Example:
#
Host to Virtual syncWe use the top-level import
field in the configuration to declare which host resources we want to sync to the virtual cluster. Each item in the import
array defines the resource via apiVersion
and kind
strings. Each apiVersion
and kind
pair can have only one entry in the import
array. The patches
field allows you to define how are certain fields of the synced resource modified before its creation(or update) in the virtual cluster.
The reversePatches
field allows you to declare how changes to certain fields of the synced resource(in this case, the one created in the virtual cluster) are propagated back to the original resource in the host cluster. Only the fields referenced in the copyFromObject
reverse patch operations are propagated.
Both these fields follow the same syntax, as documented in the "Patch syntax" chapter of this doc.
Example:
info
The sync from Host to Virtual cluster is supported only in "Multi-namespace mode"
#
Patch syntaxThe patch defines how will the vcluster behave when syncing each resource to and from the host cluster. Generally, a patch is defined by the field path
and op
(operation) that should be performed on said field.
An array of conditions
may also be set, and in such case, the field value will be modified by a patch only if the field value matches all the conditions.
Some operation types may utilize additional fields, and these will be explained in the next chapter.
Patch operations
op | Support | Description |
---|---|---|
copyFromObject | all | Copy value of the field referenced in the fromPath from the originating object to the path field of the destination object. The fromPath can be omitted, in such case, it will default to the same field path as referenced in the path . |
add | all | Add contents of the value into the path field. The value can be either scalar or a complex object. |
replace | all | Replace the contents of the path field with the contents of the value . The value can be either scalar or a complex object. |
remove | all | Remove the contents of the path field |
rewriteName | V->H | Replaces the contents of the path field with transformed content based on the namespace of the synced resource. This is typically done on the fields that refer to a resource name, and on the .metadata.name as well(implicit). This is done to avoid naming collisions when syncing resources to the host cluster, but it is not necessary when using the "Multi-namespace mode".As an example, the "logstash" value of a resource in the "logging" namespace of the vcluster named "vc" is rewritten to "logstash-x-logging-x-vc". If the resulting length of the value would be over 63 characters, the last 10 characters will be replaced with a hash of the full value. |
rewriteName + namePath + namespacePath | V->H | Similar to rewriteName , but with an addition of the namePath and/or namespacePath . This is used when a field of the synced resource is referencing a different resource via namespace and name via two separate fields. When using this option you would set the path to reference a field that is a common parent of both namePath and namespacePath , and these two fields would then contain just the relative path. For example, path: spec.includes + namePath: name + namespacePath: namespace for a resource that contains name in spec.includes.name and namespace in spec.includes.namespace . |
rewriteName + regex | V->H | Similar to rewriteName , but with an addition of the regex option for the patch. This is used when a string contains not just the resource name, but optionally a namespace, and other characters. For example, a string containing "namespace/name" can be correctly rewritten with the addition of this configuration option - regex: "$NAMESPACE/$NAME" . The vcluster uses Go regular expressions to recognize the name part with the "NAME" capture group (can be written as $NAME ), and the namespace with the "NAMESPACE" capture group (can be written as $NAMESPACE ). |
rewriteLabelKey | V->H | The keys of the .metadata.labels of the synced resources are rewritten by vcluster and plugins. This patch type allows you to rewrite the key references in the same way, so the fields that are referencing labels will still reference correct labels in their rewritten form. For example, the label key-value pair "app: curl" is rewritten to "vcluster.loft.sh/label-vcluster-x-a172cedcae: curl", so with this patch operation you can rewrite a field that contains "app" to "vcluster.loft.sh/label-vcluster-x-a172cedcae, and the controllers operating on the synced resources will work with this label just as expected. This is not necessary when using the ["Multi-namespace mode"].(#multi-namespace-mode). |
rewriteLabelSelector | V->H | This operation exists for the same reasons as described for the rewriteLabelKey operation. It is intended to be used for the key-value map fields that represent a label selector. This patch operation will rewrite all keys in the field referenced by path to the expected format for the label keys, and it will also add additional key-value pairs(with virtual namespace and vcluster name) to avoid naming conflicts. This is not necessary when using the ["Multi-namespace mode"]. |
rewriteLabelExpressionsSelector | V->H | Similar to the rewriteLabelSelector , but expects path reference a field with the matchLabels and matchExpressions sub-fields, which will have the label keys rewritten just as described for rewriteLabelKey . This is not necessary when using the ["Multi-namespace mode"]. |
V->H - patch operation is supported only for patches, or reverse patches, that are executed in the virtual to host direction.
Example of various patch operations
The example below is a useful configuration for pushing service metrics into a Prometheus instance installed in the host cluster.
You can refer to ServiceMonitor spec documentation to understand the example better.
Patch conditions
A patch can be applied conditionally by populating the conditions
field of the patch definition. The condition will be checked either on a field referenced relative to the patch path
via the subPath
field of the condition or referenced as an absolute path in the synced resource via the path
field of the condition.
A condition can have one of these three fields set to check the value respectively:
equal
- either a scalar value or a complex object can be usednotEqual
- either a scalar value or a complex object can be usedempty
- a boolean (true or false) value Examples:
#
More ExamplesA list of sample configurations can be found here - vcluster generic-sync-examples
#
Multi-namespace modeIn this mode vcluster diverges from the architecture described previously. By default, all namespaced resources that need to be synced to the host cluster are created in the namespace where vcluster is installed. But in multi-namespace mode vcluster will create a namespace in the host cluster for each namespace in the virtual cluster. The namespace name is modified to avoid conflicts between multiple vcluster instances in the same host, but the synced namespaced resources are created with the same name as in the virtual cluster. To enable this mode use the following helm value:
This mode must be enabled during vcluster creation.
Enabling, or disabling, it on an existing vcluster instance will force it into an inconsistent state.
Alpha feature
Multi-namespace mode is currently in an alpha state. This is an advanced feature that requires more permissions in the host cluster, and as a result, it can potentially cause significant disruption in the host cluster.