Plugins
A plugin runs with the same permissions as vCluster itself does in the Kubernetes cluster. Plugins can also define additional permissions, so make sure you only install plugins you trust.
Plugins overview
Plugins extend the capabilities of vCluster. They let you to add custom functionality, such as:
- Syncing specific resources from or to the virtual clusters, including cluster scoped resources like cluster roles
- Syncing custom resources from or to the virtual cluster
- Deploying resources on virtual cluster startup, such as CRDs, applications, etc.
- Manage resources and applications inside the host or virtual cluster
- Enforcing certain restrictions on synced resources or extending the existing syncers of vCluster
- Any other operator use case that could benefit from having access to the virtual cluster and the host cluster simultaneously.
A plugin in its purest form is a Kubernetes operator that has access to both the virtual cluster and the host cluster simultaneously. This is the main difference between a vCluster plugin and a regular Kubernetes operator that you would install inside the virtual cluster itself. Given this dual access, the plugin is able to translate resources between both clusters, which is the basic building block of how vCluster works.
In order to better understand how vCluster plugins work, read about Kubernetes operators and controllers.
Architecture
Each plugin runs as a binary inside the vCluster syncer container. It is by default copied over from an init container. This is done to allow easier communication between vCluster and the plugins as well as provide certain capabilities such as high-availability out of the box. The plugin itself uses go-plugin, which is used by many popular open-source projects to extend their functionality. The vCluster syncer calls the plugin binary during startup and passes relevant information to the plugin. The plugin controllers are then started with these credentials, similar to how vCluster itself starts its resource syncers.
Plugin controllers
Resource syncing enables the virtual cluster to behave like an actual Kubernetes cluster. A Kubernetes controller that is responsible for resource syncing in vCluster is called a syncer. This controller reacts on changes to objects within the virtual cluster and on changes to objects within the host cluster. The syncer tries to map each virtual object to a physical object in the host cluster and then compares those. After it discovers a change, the syncer ensures that the virtual cluster object and the physical cluster object are aligned in the desired state, and if not, the syncer changes either one of those objects to reflect the desired state.
Each plugin can define several of those resource syncers that would work exactly like the built-in syncers of vCluster. However, you don't need to sync every Kubernetes resource to the host cluster, as some can stay purely virtual. Only resources that influence the workloads need to be synced, for example, pods, services, and endpoints, while others such as deployments, replicasets, namespaces etc. are only relevant to the Kubernetes control plane and hence are not needed in the host cluster.
There are sometimes also cases where you want to manage specific core resources yourself without interfering with what vCluster is syncing, for example special secrets or configmaps that were created from the host cluster or a different resource inside the host cluster.
For this use case you can label resources vCluster should ignore either on the physical or virtual side with a label vcluster.loft.sh/controlled-by and a custom value of your choosing. This tells vCluster to ignore the resource in its syncers.
Plugin hooks
Plugin hooks are a great feature to adjust current syncing behaviour of vCluster without the need to override an already existing syncer in vCluster completely. They allow you to change outgoing objects of vCluster similar to an mutating admission controller in Kubernetes. Requirement for an hook to work correctly is that vCluster itself would sync the resource, so hooks only work for the core resources that are synced by vCluster such as pods, services, secrets etc.
If a plugin registers a hook to a specific resource, vCluster forwards all requests that match the plugin's defined hooks to the plugin and the plugin can then adjust or even deny the request completely. This opens up a wide variety of adjustment possibilities for plugins, where you for example only want to add a custom label or annotation.
Plugin SDK
If you want to start developing your own vCluster plugins, read about controller-runtime and kube builder, which uses the controller runtime internally.
vCluster provides an SDK for writing plugin controllers that abstracts a lot of the syncer complexity away from the user, but still gives you access to the underlying structures if you need it. Internally, the vCluster SDK uses the controller-runtime project, which is used by vCluster itself to create the controllers. The vCluster SDK lets you write custom plugin controllers with just a few lines of code.
Since the plugin SDK interfaces are mostly compatible with the vCluster syncers, you can also take a look at how those are implemented in the vCluster itself, which work in most cases the same way as if those would be implemented in a plugin. It would be even possible to reimplement all vCluster syncers in a separate plugin.
Most of the settings are for the init container from which vCluster copies the plugin. Adjust controlPlane.statefulSet.resources configuration to ensure the vCluster deployment has enough CPU and memory to also run the plugin.
plugins[*].resourcesdoes not change the syncer's resource requests.plugins[*].configcontent ends up in an environment variablePLUGIN_CONFIGthat is read by the plugin. This configuration has no specific format.
Config reference
plugins required <plugin_name>:object pro
Define which vCluster plugins to load.
plugins required <plugin_name>:object proname required string pro
Name is the name of the init-container and NOT the plugin name
name required string proimage required string pro
Image is the container image that should be used for the plugin
image required string proimagePullPolicy required string pro
ImagePullPolicy is the pull policy to use for the container image
imagePullPolicy required string proconfig required object pro
Config is the plugin config to use. This can be arbitrary config used for the plugin.
config required object prorbac required object pro
RBAC holds additional rbac configuration for the plugin
rbac required object prorole required object pro
Role holds extra virtual cluster role permissions for the plugin
role required object proextraRules required object[] pro
ExtraRules are extra rbac permissions roles that will be added to role or cluster role
extraRules required object[] proverbs required string[] pro
Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '*' represents all verbs.
verbs required string[] proapiGroups required string[] pro
APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of
the enumerated resources in any API group will be allowed. "" represents the core API group and "*" represents all API groups.
apiGroups required string[] proresources required string[] pro
Resources is a list of resources this rule applies to. '*' represents all resources.
resources required string[] proresourceNames required string[] pro
ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed.
resourceNames required string[] prononResourceURLs required string[] pro
NonResourceURLs is a set of partial urls that a user should have access to. *s are allowed, but only as the full, final step in the path
Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding.
Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"), but not both.
nonResourceURLs required string[] proclusterRole required object pro
ClusterRole holds extra virtual cluster cluster role permissions required for the plugin
clusterRole required object proextraRules required object[] pro
ExtraRules are extra rbac permissions roles that will be added to role or cluster role
extraRules required object[] proverbs required string[] pro
Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '*' represents all verbs.
verbs required string[] proapiGroups required string[] pro
APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of
the enumerated resources in any API group will be allowed. "" represents the core API group and "*" represents all API groups.
apiGroups required string[] proresources required string[] pro
Resources is a list of resources this rule applies to. '*' represents all resources.
resources required string[] proresourceNames required string[] pro
ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed.
resourceNames required string[] prononResourceURLs required string[] pro
NonResourceURLs is a set of partial urls that a user should have access to. *s are allowed, but only as the full, final step in the path
Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding.
Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"), but not both.
nonResourceURLs required string[] procommand required string[] pro
Command is the command that should be used for the init container
command required string[] proargs required string[] pro
Args are the arguments that should be used for the init container
args required string[] prosecurityContext required object pro
SecurityContext is the container security context used for the init container
securityContext required object proresources required object pro
Resources are the container resources used for the init container
resources required object provolumeMounts required object[] pro
VolumeMounts are extra volume mounts for the init container
volumeMounts required object[] pro