Development tutorial

In this tutorial we will implement a ConfigMap syncer. Vcluster syncs ConfigMaps out of the box, but only those that are used by one of the pods created in vcluster. Here we will have a step-by-step look at a plugin implementation that will synchronize all ConfigMaps using the vcluster plugin SDK.

Prerequisites

Before starting to develop, make sure you have installed the following tools on your computer:

docker
kubectl with a valid kube context configured
helm, which is used to deploy vcluster and the plugin
vcluster CLI v0.9.1 or higher
Go programming language build tools

Implementation

Check out the vcluster plugin example via:

git clone https://github.com/loft-sh/vcluster-plugin-example.git

You'll see a bunch of files already created, but lets take a look at the main.go file:

package main

import (
    "github.com/loft-sh/vcluster-sdk/plugin"
    "github.com/loft-sh/vcluster-sync-all-configmaps/syncers"
)

func main() {
    ctx := plugin.MustInit("sync-all-configmaps-plugin")
    plugin.MustRegister(syncers.NewConfigMapSyncer(ctx))
    plugin.MustStart()
}

Let's break down what is happening in the main() function above.

ctx := plugin.MustInit("sync-all-configmaps-plugin") - SDK will contact the vcluster backend server and retrieve it's configuration. The returned struct of type RegisterContext contains information about vcluster flags, namespace, vcluster client config, controller manager objects, etc.

plugin.MustRegister(syncers.NewConfigMapSyncer(ctx)) - we will implement the NewConfigMapSyncer function below, but for now, all we need to know is that it should return a struct that implements Base interface, which is accepted by the MustRegister function. We should call MustRegister function for each syncer that we wish to be managed by the plugins controller manager.

plugin.MustStart() - this blocking function will wait until the vcluster pod where this plugin container is running becomes the leader. Next, it will call the Init() and RegisterIndices() functions on the syncers that implement the Initializer and IndicesRegisterer respectively. Afterwards, the SDK will start its controller managers and call the RegisterSyncer or RegisterFakeSyncer function on the syncers that implement FakeSyncer and Syncer interfaces. Additionally, after configuring the default controller for the syncers, the ModifyController function is called for the syncers that implement ControllerModifier interface, which gives a plugin developer a chance to interact with the controller builder object. All these interfaces act like hooks into different points of the SDK to allow you to customize the controller that will call your syncer based on the changes to the watched resources.

Implementing a syncer for a namespaced resource

In this chapter, we take a look at the sync-all-configmaps.go file that can be found in the syncer directory.

package syncers

import (
    "github.com/loft-sh/vcluster-sdk/syncer"
    syncercontext "github.com/loft-sh/vcluster-sdk/syncer/context"
    "github.com/loft-sh/vcluster-sdk/syncer/translator"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/equality"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func NewConfigMapSyncer(ctx *syncercontext.RegisterContext) syncer.Syncer {
    return &configMapSyncer{
        NamespacedTranslator: translator.NewNamespacedTranslator(ctx, "configmap", &corev1.ConfigMap{}),
    }
}

type configMapSyncer struct {
    translator.NamespacedTranslator
}

After an import block, we see the NewConfigMapSyncer function, which is being called from the main.go. It returns a new instance of the configMapSyncer struct, which is defined by a single nested anonymous struct of type NamespacedTranslator. The NamespacedTranslator implements many functions of the Syncer interface for us, and we will implement the remaining ones - SyncDown and Sync.

info

You can get more familiar with the interfaces mentioned above by reading the SDK source files on GitHub - vcluster-sdk/syncer/types.go and vcluster-sdk/syncer/translator/translator.go, or by using pkg.go.dev website - Syncer and NamespacedTranslator.

The SyncDown function mentioned above is called by the vcluster SDK when a given resource, e.g. a ConfigMap, is created in the vcluster, but it doesn't exist in the host cluster yet. To create a ConfigMap in the host cluster we will call the SyncDownCreate function with the output of the translate function as third parameter. This demonstrates a typical pattern used in the vcluster syncer implementations.

func (s *configMapSyncer) SyncDown(ctx *syncercontext.syncercontext, vObj client.Object) (ctrl.Result, error) {
    return s.SyncDownCreate(ctx, vObj, s.translate(vObj.(*corev1.ConfigMap)))
}

func (s *configMapSyncer) translate(vObj client.Object) *corev1.ConfigMap {
    return s.TranslateMetadata(vObj).(*corev1.ConfigMap)
}

The TranslateMetadata function used above produces a ConfigMap object that will be created in the host cluster. It is a deep copy of the ConfigMap from vcluster, but with certain metadata modifications - the name and labels are transformed, some vcluster labels and annotations are added, many metadata fields are stripped (uid, resourceVersion, etc.).

Next, we need to implement code that will handle the updates of the ConfigMap. When a ConfigMap in vcluster or host cluster is updated, the vcluster SDK will call the Sync function of the syncer. Current ConfigMap resource from the host cluster and from vcluster are passed as the second and third parameters respectively. In the implementation below, you can see another pattern used by the vcluster syncers. The translateUpdate function will return nil when no change to the ConfigMap in the host cluster is needed, and the SyncDownUpdate function will not do an unnecessary update API call in such case.

func (s *configMapSyncer) Sync(ctx *syncercontext.SyncContext, pObj client.Object, vObj client.Object) (ctrl.Result, error) {
    return s.SyncDownUpdate(ctx, vObj, s.translateUpdate(pObj.(*corev1.ConfigMap), vObj.(*corev1.ConfigMap)))
}

func (s *configMapSyncer) translateUpdate(pObj, vObj *corev1.ConfigMap) *corev1.ConfigMap {
    var updated *corev1.ConfigMap

    changed, updatedAnnotations, updatedLabels := s.TranslateMetadataUpdate(vObj, pObj)
    if changed {
        updated = translator.NewIfNil(updated, pObj)
        updated.Labels = updatedLabels
        updated.Annotations = updatedAnnotations
    }

    // check if the data has changed
    if !equality.Semantic.DeepEqual(vObj.Data, pObj.Data) {
        updated = translator.NewIfNil(updated, pObj)
        updated.Data = vObj.Data
    }

    // check if the binary data has changed
    if !equality.Semantic.DeepEqual(vObj.BinaryData, pObj.BinaryData) {
        updated = translator.NewIfNil(updated, pObj)
        updated.BinaryData = vObj.BinaryData
    }
    return updated
}

As you might have noticed, the changes to the Immutable field of the ConfigMap are not being checked and propagated to the updated ConfigMap. That is done just for the simplification of the code in this tutorial. In the real world use cases, there will likely be many scenarios and edge cases that you will need to handle differently than just with a simple comparison and assignment. For example, you will need to look out for label selectors that are interpreted in the host cluster, e.g. pod selectors in the NetworkPolicy resources are interpreted by the host cluster network plugin. Such selectors must be translated when synced down to the host resources. Several functions for the common use cases are built into the SDK in the syncer/translator package, including the TranslateLabelSelector function.

Also, notice that this example lacks the updates to the ConfigMap resource in vcluster. Here we propagate the changes only down to the ConfigMap in the host cluster, but there are resources or use cases where a syncer would update the synced resource in vcluster. For example, this might be an update of the status subresource or synchronization of any other field that some controller sets on the host side, e.g., finalizers. Implementation of such updates needs to be considered on case-by-case basis. For some use cases, you may need to sync the resources in the opposite direction, from the host cluster up into the vcluster, or even in both directions. If that is what your plugin needs to do, you will implement the UpSyncer interface defined by the SDK.

Adding a hook for changing a resource on the fly

Hooks are a great feature to adjust current syncing behaviour of vcluster without the need to override an already existing syncer in vcluster completely. They allow you to change outgoing objects of vcluster similar to an mutating admission controller in Kubernetes. Requirement for an hook to work correctly is that vcluster itself would sync the resource, so hooks only work for the core resources that are synced by vcluster such as pods, services, secrets etc.

To add a hook to your plugin, you simply need to create a new struct that implements the ClientHook interface:

package myhook

import (
    "context"
    "fmt"
    "github.com/loft-sh/vcluster-sdk/hook"
    corev1 "k8s.io/api/core/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func NewPodHook() hook.ClientHook {
    return &podHook{}
}

type podHook struct{}

func (p *podHook) Name() string {
    return "pod-hook"
}

func (p *podHook) Resource() client.Object {
    return &corev1.Pod{}
}

The Name() function defines the name of the hook which is used for logging purposes. The Resource() function returns the object you want to mutate. Besides those functions you can now define what actions you want to hook into inside vcluster's syncer:

type MutateCreateVirtual interface {
    MutateCreateVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateUpdateVirtual interface {
    MutateUpdateVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateDeleteVirtual interface {
    MutateDeleteVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateGetVirtual interface {
    MutateGetVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateCreatePhysical interface {
    MutateCreatePhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateUpdatePhysical interface {
    MutateUpdatePhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateDeletePhysical interface {
    MutateDeletePhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateGetPhysical interface {
    MutateGetPhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

By implementing one or more of the above interfaces you will receive events from vcluster that allows you to mutate an outgoing or incoming object to vcluster. For example, to add an hook that adds a custom label to a pod, you can add the following code:

var _ hook.MutateCreatePhysical = &podHook{}

func (p *podHook) MutateCreatePhysical(ctx context.Context, obj client.Object) (client.Object, error) {
    pod, ok := obj.(*corev1.Pod)
    if !ok {
        return nil, fmt.Errorf("object %v is not a pod", obj)
    }

    if pod.Labels == nil {
        pod.Labels = map[string]string{}
    }
    pod.Labels["created-by-plugin"] = "pod-hook"
    return pod, nil
}

var _ hook.MutateUpdatePhysical = &podHook{}

func (p *podHook) MutateUpdatePhysical(ctx context.Context, obj client.Object) (client.Object, error) {
    pod, ok := obj.(*corev1.Pod)
    if !ok {
        return nil, fmt.Errorf("object %v is not a pod", obj)
    }

    if pod.Labels == nil {
        pod.Labels = map[string]string{}
    }
    pod.Labels["created-by-plugin"] = "pod-hook"
    return pod, nil
}

Incoming objects into vcluster can be modified through the MutateGetPhysical or MutateGetVirtual which allows you to change how vcluster is retrieving objects from either the virtual or physical cluster. This can be useful if you don't want vcluster to change something you have mutated back for example.

Build and push your plugin

Now you can run docker commands to build your container image and push it to the registry. docker build -t your_org/vcluster-sync-all-configmaps . && docker push your_org/vcluster-sync-all-configmaps

Add plugin.yaml

The last step before installing your plugin is creating a yaml file with your plugin metadata. This file follows the format of the Helm values files. It will be merged with other values files when a vcluster is installed or upgraded. For the plugin we just implemented and built it would look like this:

plugin:
  sync-all-configmaps-plugin:
    image: your_org/vcluster-sync-all-configmaps
syncer:
  extraArgs:
    - "--sync=-configmaps"

The first three lines contain a minimal definition of a vcluster plugin - a container name based on the key (second line) and container image (third line). The last three lines then contain extra values that the plugin will apply to the vcluster chart. These are needed for this particular plugin and are not mandatory otherwise. Our plugin would be syncing some ConfigMaps that would also be synced by the built-in "configmaps" syncer of the vcluster, and to avoid conflicting updates we will disable the built-in syncer by passing an additional command-line argument to the syncer container.

Deploy the plugin

You can deploy your plugin to a vcluster using the same commands as described on the overview page, for example, with the vcluster CLI.

vcluster create my-vcluster -n my-vcluster -f plugin.yaml

Fast Plugin Development with DevSpace

When developing your plugin we recommend using the devspace CLI tool for running your local plugin source code directly in Kubernetes. The appropriate configuration is already present in the devspace.yaml and you can start developing by running the following command:

info

If you want to develop within a remote Kubernetes cluster (as opposed to docker-desktop or minikube), make sure to exchange PLUGIN_IMAGE in the devspace.yaml with a valid registry path you can push to.

After successfully setting up the tools, start the development environment with:

devspace dev -n vcluster

After a while a terminal should show up with additional instructions. Enter the following command to start the plugin:

go run -mod vendor main.go

The output should look something like this:

I0124 11:20:14.702799    4185 logr.go:249] plugin: Try creating context...
I0124 11:20:14.730044    4185 logr.go:249] plugin: Waiting for vcluster to become leader...
I0124 11:20:14.731097    4185 logr.go:249] plugin: Starting syncers...
[...]
I0124 11:20:15.957331    4185 logr.go:249] plugin: Successfully started plugin.

You can now change a file locally in your IDE and then restart the command in the terminal to apply the changes to the plugin.

DevSpace will create a development vcluster which will execute your plugin. Any changes made within the vcluster created by DevSpace will execute against your plugin.

vcluster list                

 NAME       NAMESPACE       STATUS    CONNECTED   CREATED                          AGE
 vcluster   vcluster        Running   True        2022-09-06 20:33:20 +1000 AEST   2h26m8s

After you are done developing or you want to recreate the environment, delete the development environment with:

devspace purge -n vcluster

Prerequisites​

Implementation​

Implementing a syncer for a namespaced resource​

Adding a hook for changing a resource on the fly​

Build and push your plugin​

Add plugin.yaml​

Deploy the plugin​

Fast Plugin Development with DevSpace​