Development tutorial

In this tutorial we will implement a Car syncer. Here we will have a step-by-step look at a plugin implementation that will synchronize all custom car objects using the vCluster plugin SDK.

info

You can find other examples in the vCluster SDK Repository

Prerequisites

Before starting to develop, make sure you have installed the following tools on your computer:

docker
kubectl with a valid kube context configured
helm, which is used to deploy vCluster and the plugin
vcluster CLI v0.9.1 or higher
Go programming language build tools

Implementation

Check out the vCluster plugin example via:

git clone https://github.com/loft-sh/vcluster-plugin-example.git

You'll see a bunch of files already created, but lets take a look at the main.go file:

package main

import (
    "github.com/loft-sh/vcluster-plugin-example/syncers"
    "github.com/loft-sh/vcluster-sdk/plugin"
)

func main() {
    ctx := plugin.MustInit()
    plugin.MustRegister(syncers.NewCarSyncer(ctx))
    plugin.MustStart()
}

Let's break down what is happening in the main() function above.

ctx := plugin.MustInit() - SDK will init the plugin and retrieve configuration from the vCluster syncer. The returned struct contains information about vCluster flags, namespace, vCluster client config, controller manager objects, etc.

plugin.MustRegister(syncers.NewCarSyncer(ctx)) - we will implement the NewCarSyncer function below, but for now, all we need to know is that it should return a struct that implements an interface which is accepted by the MustRegister function.

plugin.MustStart() - this blocking function will wait until the vCluster pod where this plugin is running becomes the leader.

Implementing a syncer for a namespaced resource

In this chapter, we take a look at the car.go file that can be found in the syncer directory.

package syncers

import (
    "context"
    "os"

    examplev1 "github.com/loft-sh/vcluster-plugin-example/apis/v1"
    synccontext "github.com/loft-sh/vcluster/pkg/controllers/syncer/context"
    "github.com/loft-sh/vcluster/pkg/controllers/syncer/translator"
    "github.com/loft-sh/vcluster/pkg/scheme"
    synctypes "github.com/loft-sh/vcluster/pkg/types"
    "github.com/loft-sh/vcluster/pkg/util"
    "github.com/loft-sh/vcluster/pkg/util/translate"
    "k8s.io/apimachinery/pkg/api/equality"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func init() {
    // Make sure our scheme is registered
    _ = examplev1.AddToScheme(scheme.Scheme)
}

func NewCarSyncer(ctx *synccontext.RegisterContext) synctypes.Base {
    return &carSyncer{
        NamespacedTranslator: translator.NewNamespacedTranslator(ctx, "car", &examplev1.Car{}),
    }
}

type carSyncer struct {
    translator.NamespacedTranslator
}

After an import block, we see the NewCarSyncer function, which is being called from the main.go. It returns a new instance of the carSyncer struct, which is defined by a single nested anonymous struct of type NamespacedTranslator. The NamespacedTranslator implements many functions of the Syncer interface for us, and we will implement the remaining ones - SyncDown and Sync.

The SyncDown function mentioned above is called by the vCluster SDK when a given resource, e.g. a Car, is created in the vCluster, but it doesn't exist in the host cluster yet. To create a ConfigMap in the host cluster we will call the SyncToHostCreate function with the output of the translate function as third parameter. This demonstrates a typical pattern used in the vCluster syncer implementations.

func (s *carSyncer) SyncToHost(ctx *synccontext.SyncContext, vObj client.Object) (ctrl.Result, error) {
    return s.SyncToHostCreate(ctx, vObj, s.TranslateMetadata(ctx.Context, vObj).(*examplev1.Car))
}

func (s *carSyncer) Sync(ctx *synccontext.SyncContext, pObj client.Object, vObj client.Object) (ctrl.Result, error) {
    return s.SyncToHostUpdate(ctx, vObj, s.translateUpdate(ctx.Context, pObj.(*examplev1.Car), vObj.(*examplev1.Car)))
}

The TranslateMetadata function used above produces a Car object that will be created in the host cluster. It is a deep copy of the Car from vCluster, but with certain metadata modifications - the name and labels are transformed, some vCluster labels and annotations are added, many metadata fields are stripped (uid, resourceVersion, etc.).

Next, we need to implement code that will handle the updates of the Car. When a CAr in vCluster or host cluster is updated, the vCluster SDK will call the Sync function of the syncer. Current Car resource from the host cluster and from vCluster are passed as the second and third parameters respectively. In the implementation below, you can see another pattern used by the vCluster syncers. The translateUpdate function will return nil when no change to the Car in the host cluster is needed, and the SyncToHostUpdate function will not do an unnecessary update API call in such case.

func (s *carSyncer) Sync(ctx *synccontext.SyncContext, pObj client.Object, vObj client.Object) (ctrl.Result, error) {
    return s.SyncToHostUpdate(ctx, vObj, s.translateUpdate(ctx.Context, pObj.(*examplev1.Car), vObj.(*examplev1.Car)))
}

func (s *carSyncer) translateUpdate(ctx context.Context, pObj, vObj *examplev1.Car) *examplev1.Car {
    var updated *examplev1.Car

    // check annotations & labels
    changed, updatedAnnotations, updatedLabels := s.TranslateMetadataUpdate(ctx, vObj, pObj)
    if changed {
        updated = translator.NewIfNil(updated, pObj)
        updated.Labels = updatedLabels
        updated.Annotations = updatedAnnotations
    }

    // check spec
    if !equality.Semantic.DeepEqual(vObj.Spec, pObj.Spec) {
        updated = translator.NewIfNil(updated, pObj)
        updated.Spec = vObj.Spec
    }

    return updated
}

Here we propagate the changes only down to the Car in the host cluster, but there are resources or use cases where a syncer would update the synced resource in vCluster. For example, this might be an update of the status subresource or synchronization of any other field that some controller sets on the host side, e.g., finalizers. Implementation of such updates needs to be considered on case-by-case basis. For some use cases, you may need to sync the resources in the opposite direction, from the host cluster up into the vCluster, or even in both directions. If that is what your plugin needs to do, you will implement the UpSyncer interface defined by the SDK.

Adding a hook for changing a resource on the fly

Hooks are a great feature to adjust current syncing behaviour of vCluster without the need to override an already existing syncer in vCluster completely. They allow you to change outgoing objects of vCluster similar to an mutating admission controller in Kubernetes. Requirement for an hook to work correctly is that vCluster itself would sync the resource, so hooks only work for the core resources that are synced by vCluster such as pods, services, secrets etc.

To add a hook to your plugin, you simply need to create a new struct that implements the ClientHook interface:

package hooks

import (
    "context"
    "fmt"

    "github.com/loft-sh/vcluster-sdk/plugin"
    corev1 "k8s.io/api/core/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func NewPodHook() plugin.ClientHook {
    return &podHook{}
}

type podHook struct{}

func (p *podHook) Name() string {
    return "pod-hook"
}

func (p *podHook) Resource() client.Object {
    return &corev1.Pod{}
}

The Name() function defines the name of the hook which is used for logging purposes. The Resource() function returns the object you want to mutate. Besides those functions you can now define what actions you want to hook into inside vCluster's syncer:

type MutateCreateVirtual interface {
    MutateCreateVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateUpdateVirtual interface {
    MutateUpdateVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateDeleteVirtual interface {
    MutateDeleteVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateGetVirtual interface {
    MutateGetVirtual(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateCreatePhysical interface {
    MutateCreatePhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateUpdatePhysical interface {
    MutateUpdatePhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateDeletePhysical interface {
    MutateDeletePhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

type MutateGetPhysical interface {
    MutateGetPhysical(ctx context.Context, obj client.Object) (client.Object, error)
}

By implementing one or more of the above interfaces you will receive events from vCluster that allows you to mutate an outgoing or incoming object to vCluster. For example, to add an hook that adds a custom label to a pod, you can add the following code:

var _ plugin.MutateCreatePhysical = &podHook{}

func (p *podHook) MutateCreatePhysical(ctx context.Context, obj client.Object) (client.Object, error) {
    pod, ok := obj.(*corev1.Pod)
    if !ok {
        return nil, fmt.Errorf("object %v is not a pod", obj)
    }

    if pod.Labels == nil {
        pod.Labels = map[string]string{}
    }
    pod.Labels["created-by-plugin"] = "pod-hook"
    return pod, nil
}

var _ plugin.MutateUpdatePhysical = &podHook{}

func (p *podHook) MutateUpdatePhysical(ctx context.Context, obj client.Object) (client.Object, error) {
    pod, ok := obj.(*corev1.Pod)
    if !ok {
        return nil, fmt.Errorf("object %v is not a pod", obj)
    }

    if pod.Labels == nil {
        pod.Labels = map[string]string{}
    }
    pod.Labels["created-by-plugin"] = "pod-hook"
    return pod, nil
}

Incoming objects into vCluster can be modified through the MutateGetPhysical or MutateGetVirtual which allows you to change how vCluster is retrieving objects from either the virtual or physical cluster. This can be useful if you don't want vCluster to change something you have mutated back for example.

Build and push your plugin

Now you can run docker commands to build your container image and push it to the registry. docker build -t your_org/vcluster-plugin-example . && docker push your_org/vcluster-plugin-example

Add plugin.yaml

The last step before installing your plugin is creating a yaml file with your plugin metadata. This file follows the format of the Helm values files. It will be merged with other values files when a vCluster is installed or upgraded. For the plugin we just implemented and built it would look like this:

# Plugin Definition below. This is essentially a valid helm values file that will be merged
# with the other vcluster values during vcluster create or helm install.
plugin:
  vcluster-plugin-example:
    version: v2
    image: ghcr.io/loft-sh/vcluster-plugin-example:v1
    rbac:
      role:
        extraRules:
          - apiGroups: ["example.loft.sh"]
            resources: ["cars"]
            verbs: ["create", "delete", "patch", "update", "get", "list", "watch"]
      clusterRole:
        extraRules:
          - apiGroups: ["apiextensions.k8s.io"]
            resources: ["customresourcedefinitions"]
            verbs: ["get", "list", "watch"]

# Make sure the cluster role is enabled or otherwise the plugin won't be able to watch custom
# resource definitions.
rbac:
  clusterRole:
    create: true

Deploy the plugin

You can deploy your plugin to a vCluster using the same commands as described on the overview page, for example, with the vCluster CLI.

vcluster create my-vcluster -n my-vcluster -f plugin.yaml

Fast Plugin Development with DevSpace

When developing your plugin we recommend using the devspace CLI tool for running your local plugin source code directly in Kubernetes. The appropriate configuration is already present in the devspace.yaml and you can start developing by running the following command:

After successfully setting up the tools, start the development environment with:

devspace dev -n vcluster

After a while a terminal should show up with additional instructions. Enter the following command to start the plugin:

go build -mod vendor -o plugin main.go && /vcluster/syncer start

You can now change a file locally in your IDE and then restart the command in the terminal to apply the changes to the plugin.

DevSpace will create a development vCluster which will execute your plugin. Any changes made within the vCluster created by DevSpace will execute against your plugin.

vcluster list

 NAME       NAMESPACE       STATUS    CONNECTED   CREATED                          AGE
 vcluster   vcluster        Running   True        2022-09-06 20:33:20 +1000 AEST   2h26m8s

After you are done developing or you want to recreate the environment, delete the development environment with:

devspace purge -n vcluster

Prerequisites​

Implementation​

Implementing a syncer for a namespaced resource​

Adding a hook for changing a resource on the fly​

Build and push your plugin​

Add plugin.yaml​

Deploy the plugin​

Fast Plugin Development with DevSpace​