Custom resources

Supported Configurations

Running the control plane as a container with:

• Shared Nodes

vClustervClusterAn open-source software product that creates and manages tenant clusters within Kubernetes infrastructure. vCluster provides tenant isolation capabilities while reducing infrastructure costs.Related: Tenant cluster, Control plane cluster allows you to sync custom resources from the tenant clusterTenant clusterA fully isolated Kubernetes environment provisioned for a single tenant. Each tenant cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Tenant cluster to the control plane cluster. This is useful for syncing resources that are not included in the default sync behavior.

This feature works for namespace-scoped resources backed by a CustomResourceDefinition (CRD) or served through API aggregation.

If a synced custom resource creates additional resources in the control plane clusterControl plane clusterThe Kubernetes cluster that hosts the virtualized control planes for tenant clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both.Related: Tenant cluster, Control plane cluster, Tenant cluster, vCluster attempts to detect and sync those resources back to the tenant cluster. For example, a custom resource that creates a Secret has that Secret automatically synced into the tenant cluster.

For help choosing a sync direction, and for lifecycle and troubleshooting guidance, see Manage custom resources.

Automatic RBAC for configured resources

vCluster automatically adds the required cluster and namespace-level RBAC permissions for retrieving the CustomResourceDefinition and syncing the configured resources. Extension API servers can require additional permissions for related resources. See Authorization on the control plane cluster.

Only namespace-scoped resources

This feature currently only works for namespace-scoped resources.

Namespace syncing

If you want to sync many custom resources, consider using namespace syncing.

Enable custom resource syncing

To enable custom resource syncing, run kubectl api-resources against the control plane cluster to identify the resource. Use <plural>.<group> as the key in customResources. For example, use customobjects.example.io for the customobjects resource in the example.io/v1 API group.

At startup, vCluster determines how the control plane cluster serves the resource. It copies a CRD-backed resource's definition into the tenant cluster. For a resource served by an extension API serverAPI ServerThe core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects.Related: Control Plane, rate-limiting, vCluster generates a schemaless CRD instead.

You don't need to specify an API version by default. vCluster uses the storage version for a CRD-backed resource. For an aggregated API resource, it uses the API group's preferred version. To select another version, append it to the resource key in vcluster.yaml.

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true

Aggregated API resources

Not every extension type in a Kubernetes cluster is backed by a CustomResourceDefinition. Kubernetes can also serve types through API aggregation, where an extension API server registered as an APIService owns the type instead of a CRD.

vCluster syncs these resources through the same customResources configuration. You don't declare that a resource comes from an extension API server. vCluster resolves every configured resource against the control plane cluster on startup:

• If the control plane cluster has a CRD for the resource, vCluster copies that CRD into the tenant cluster.
• If it doesn't, vCluster looks the resource up in the control plane cluster's discovery data. When an extension API server serves the resource, vCluster generates a schemaless CRD for it in the tenant cluster.
• If neither a CRD nor an extension API server serves the resource, the tenant cluster fails to start.

This resolution happens only on startup. If the extension API server isn't serving the resource then, vCluster fails to start and tries again when the control plane pod restarts. An extension API server that appears or recovers later is picked up on that next restart.

Because kubectl get crds doesn't list resources that an extension API server serves, use kubectl api-resources to find them.

Configure aggregated API resource sync to host

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true

Generated CRDs for aggregated API resources

vCluster builds the generated CRD from the control plane cluster's discovery data. The kind, plural name, scope, and presence of a status subresource come from the control plane cluster. vCluster derives the singular name by lowercasing the kind and derives the list kind by appending List. It marks the CRD with the vcluster.loft.sh/generated-from: aggregated-apiserver annotation and creates or updates it when the tenant cluster starts.

Objects of a generated CRD live in the tenant cluster's own data store, like any other custom resource, so snapshots include them.

The generated CRD has no resource-specific OpenAPI schema and preserves unknown fields. The tenant API server doesn't validate custom fields against the extension API server's schema. Keep the following behaviors in mind:

• kubectl explain returns no field documentation for the resource inside the tenant cluster.
• The tenant cluster doesn't prune unknown fields.
• When the resource key includes no version, vCluster uses the API group's preferred version on the control plane cluster. For CRD-backed resources, it uses the storage version instead.
• When you change the version in the resource key, vCluster serves the new version and retains previously stored versions without serving them.

vCluster uses the vcluster.loft.sh/generated-from: aggregated-apiserver annotation as the adoption marker. If an existing CRD has this marker, vCluster adopts it and updates its specification. If a CRD of the same name doesn't have this marker, the tenant cluster fails to start. Delete that CRD or remove the resource from customResources.

Removing an entry from customResources, or setting enabled: false, stops the syncer. vCluster doesn't delete the generated CRD, its objects, or the objects it created on the control plane cluster. The tenant cluster keeps serving the resource, and writes to it stay tenant-local because nothing validates or syncs them.

Authorization on the control plane cluster

vCluster grants itself the permissions it needs for every resource listed in customResources. Some extension API servers authorize a request against a related resource that you never configure for sync. For example, an API server can check access to a grouping resource before it serves or accepts an object. Grant those permissions explicitly. This example grants access to a related resource in the vCluster control plane namespace:

Grant extra permissions for an extension API server

rbac:
  role:
    extraRules:
      - apiGroups: ["example.io"]
        resources: ["customobjectgroups"]
        verbs: ["get", "list", "watch"]

Use rbac.role.extraRules for resources in the vCluster control plane namespace. Use rbac.clusterRole.extraRules for cluster-scoped resources or namespaced resources in other control plane cluster namespaces. For details, see Add custom RBAC rules.

Synchronous validation on the control plane cluster

A generated CRD has no schema, so the tenant cluster can't validate an aggregated API resource on its own. vCluster validates it on the control plane cluster instead. When a client creates, updates, or patches such a resource, vCluster writes the change through to the extension API server before it answers the request.

• If the extension API server accepts the object, the request succeeds.
• If the extension API server rejects the object, vCluster attempts to roll the tenant write back, and returns the rejection to the client. vCluster rewrites the control plane cluster's object and namespace names in the message, details, and causes. The error refers to the tenant object.

Write-through validates the request. It doesn't replace syncing. The syncerSyncerA component in vCluster that synchronizes resources between the tenant cluster and the control plane cluster, enabling tenant clusters to function while maintaining isolation.Related: vCluster, Tenant cluster still reconciles drift in both directions, propagates deletions, and removes control plane cluster objects that an interrupted request left behind.

Key behaviors for write-through:

• Only create, update, and patch requests on the resource itself are written through. Requests to subresources aren't.
• Deletes aren't written through. The syncer removes the control plane cluster object after the tenant delete has already returned.
• A rejected create exists in the tenant cluster until the rollback removes it, so watchers can observe a create followed by a delete.
• A rejected write can stay in the tenant cluster for good. Rollback is best-effort, so a failed request isn't proof that nothing was stored.
• vCluster pins each update and patch to the object it snapshotted for rollback. When a concurrent write lands first, the request fails with 409 Conflict instead of being reverted. Retry it against the current object.
• Dry-run requests aren't written through. kubectl apply --dry-run=server validates against the tenant cluster only, so it doesn't report extension API server errors.
• vCluster rejects writes with 503 Service Unavailable until the syncer finishes registering the resource on startup. This keeps writes from skipping validation.
• vCluster rejects a write body it can't prepare for rollback with 415 Unsupported Media Type. JSON, YAML, and server-side apply bodies are supported.
• Each control planeControl PlaneThe container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. In vCluster, each tenant cluster has its own control plane components.Related: API Server, vCluster replica allows up to 50 write-through requests at a time and rejects further writes with 429 Too Many Requests. A write that the control plane cluster doesn't answer within 30 seconds fails.
• The control plane cluster's audit log attributes writes that start in the tenant cluster to the vCluster identity.

Deletion is asynchronous. The tenant delete returns as soon as the tenant cluster removes the object, and the syncer removes the control plane cluster object afterward. During that window the control plane cluster object still exists, and an extension API server that enforces policy keeps enforcing it. A successful delete in the tenant cluster isn't proof that enforcement stopped.

vCluster can't tell its own rejected write apart from a newer one. It skips the rollback when another write reaches the object first. It also gives up after a few conflicting attempts. An abandoned rollback leaves the object in the tenant cluster. The control plane cluster keeps rejecting it, so the syncer retries without ever succeeding. The vcluster_writethrough_rollback_failures_total metric counts these cases.

The control plane cluster owns the status

When an aggregated API resource has a status subresource, the status syncs from the control plane cluster to the tenant cluster only. vCluster doesn't write tenant status updates through, and the syncer doesn't push them either. Tenant status fields remain local until the same fields change on the control plane cluster.

Limitations

The generated CRD serves the resource itself, plus the status subresource when the control plane cluster serves it. Extension API servers can expose additional subresources, such as scale, logs, or exec. Those aren't available inside the tenant cluster, because a CRD can't serve them.

Only namespace-scoped resources sync to the control plane cluster, the same as for CRD-backed resources.

Versioned example

To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a /, using the format <resource>/<version> in your configuration.

The specified version must exist in the control plane cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the control plane cluster's storage version, vCluster automatically converts custom resources between the two versions.

For example, to use version v1 of the certificaterequests.cert-manager.io CRD, specify:

Configure CR sync to host

sync:
  toHost:
    customResources:
      certificaterequests.cert-manager.io/v1:
        enabled: true
        

Unique CRD versions

vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the tenant cluster fails to start.

Upgrade custom resources with explicit CRD API versions

vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the tenant cluster fails to start.

vCluster modifies the CRD inside the tenant cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the tenant cluster or change the sync configuration.

Upgrade from no version to specified version

When upgrading from an unversioned CRD to a versioned one (certificaterequests.cert-manager.io → certificaterequests.cert-manager.io/v1), vCluster checks the CRD's storage version. If the storage version is not v1, vCluster updates the CRD to include v1 as a new version in addition to the current storage version.

Upgrade from one specified version to another specified version

When upgrading between versioned CRDs (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io/v2), the CRD is updated to include the new version v2, and the old version v1 is kept as well.

Upgrade from a specified version to no version specified

When upgrading from a versioned CRD to an unversioned one (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io), the behavior depends on the host's storage version:

• If the CRD's storage version in the control plane cluster is v1, nothing happens and the CRD in the tenant cluster remains as certificaterequests.cert-manager.io/v1.
• If the storage version is not v1, vCluster updates the CRD to include the storage version as a new version in addition to v1.

Patches

Use sync.toHost.customResources.<resource>.patches to transform custom resource fields while syncing to the control plane cluster. Custom resources support expression patches, reference patches, and labels patches. See Patching synced resources for the full syntax and directionality rules.

Reference patches are useful when a custom resource field points to another synced resource, such as a Secret:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.secretName
          reference:
            apiVersion: v1
            kind: Secret

Expression patches are useful when the tenant and control plane cluster need different field values:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.hosts[*]
          expression: "value.startsWith('prod-') ? value : `prod-${value}`"
          reverseExpression: "value.startsWith('prod-') ? value.slice(5) : value"

A selector held as an opaque string can't be translated, so vCluster copies it verbatim. In the default single-namespace sync mode, every tenant namespace maps to the vCluster namespace on the control plane cluster. A copied selector can therefore match objects the policy was never written for. An expression patch can scope the field to the namespace it came from. For a worked example, see Scope a synced selector field to its tenant.

Sync an Istio waypoint Gateway using custom resources

Use native Gateway API sync for core Gateway API resources

For syncing core Kubernetes Gateway API resources (Gateway, HTTPRoute, TLSRoute, ReferenceGrant, BackendTLSPolicy), use native Gateway API sync instead of custom-resource syncing. The example below covers Istio waypoint Gateways using the HBONE listener, which is outside the standard Gateway API channel and still requires the custom-resource path.

Install the Gateway API CRDs in the control plane cluster first. See Create waypoint proxy in the host for the CRD installation step.

To-host sync goes from the tenant cluster to the control plane cluster, so create the waypoint Gateway in the tenant cluster instead of the control plane cluster. Enable sync for the resource:

Sync an Istio waypoint Gateway to the control plane cluster

sync:
  toHost:
    customResources:
      gateways.gateway.networking.k8s.io:
        enabled: true

Then create the same Gateway manifest shown in the Istio integration guide, but apply it to the tenant cluster. Point kubectl at your tenant cluster's context, not the control plane cluster context used in the linked step, and add -n if the workloads it routes for aren't in the current namespace:

waypoint-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Create Waypoint Gateway in the tenant cluster

kubectl --context="<tenant-cluster-context>" create -f waypoint-gateway.yaml

vCluster syncs the Gateway to the control plane cluster, where Istio's waypoint controller reconciles it into a running proxy.

Config reference

customResources {key: object}

CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work.

enabled required boolean

Enabled defines if this option should be enabled.

scope string

Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported.

patches object[]

Patches patch the resource according to the provided specification.

path required string

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression string

Expression transforms the value according to the given JavaScript expression.

reverseExpression string

ReverseExpression transforms the value according to the given JavaScript expression.

reference object

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string

APIVersion is the apiVersion of the referenced object.

apiVersionPath string

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string

Kind is the kind of the referenced object.

kindPath string

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath string

NamePath is the optional relative path to the reference name within the object.

namespacePath string

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels object

Labels treats the path value as a labels selector.