Skip to main content
Version: main 🚧

Manage tenant resources with GitOps

When Argo CD or Flux applies Platform resources from Git, each write is checked twice. Kubernetes RBAC or the platform's permissions decide whether the sync identity may write the resource. Then the platform holds the write to the limits of the tenant that owns the resource. This page covers which identity to use, and the labels and names a tenant's manifests need. For the labels and name rules themselves, see the tenant boundary reference.

Before you begin​

  • Create the tenant and grant it the templates, control plane clusters, and other inventory its projects use. A write inside a tenant's project is always limited to that tenant's grants, no matter who sends it. See Create and configure a tenant.
  • Connect your GitOps tool to the platform's management API. The GitOps end-to-end guide shows Argo CD applying management.loft.sh resources with its own service account.
  • Ownership is fixed at the time a resource is created. Adding, changing, or removing a resource's tenant.vcluster.com/owner label later is refused, so put each resource in the right tenant from its first sync.

Choose the identity​

The identity your GitOps tool uses needs permission to write each resource. For tenant-owned resources, the platform also checks the owning tenant's limits.

IdentityPlatform-owned resourcesA tenant's resources
The tool's Kubernetes service account, or another control plane cluster identityWith the required RBAC permissions on the management API.Updates to existing cluster-scoped resources, and writes inside a tenant's project, with the required RBAC permissions. Can't create cluster-scoped resources, such as projects.
A platform admin user's access keyWithin that user's permissions.Creates and updates. Writes are restricted to what the tenant may do.
A tenant user's access keyNo.Within both the tenant and the user's permissions.

A Kubernetes identity isn't a Platform user, but it doesn't need one to update a tenant's existing projects, users, or teams. Its RBAC permissions still apply. An update uses the stored resource's owner, and a write inside a project uses that project's tenant.

A Kubernetes identity can't create a tenant-owned cluster-scoped resource. The platform refuses the owner label from it, and refuses a <tenant>-- name without that label. Create a tenant's projects, users, and teams with an access key, then let the service account manage them.

These writes can't exceed the tenant's grants or bypass restrictions on roles and resource kinds. A sync identity with access to several tenants is limited to each resource's owning tenant, rather than to one tenant selected for the sync.

If you choose an access key, create a dedicated user for the tool, as Users and teams recommends. A sync that creates resources for several tenants needs a platform admin user. A tenant user's key confines the sync to that tenant. Keep platform-owned manifests in a separate sync with a platform identity.

Write the manifests​

What goes in the manifests depends on the identity that applies them. Customize the tenant name and project suffix for the examples on this page.

Modify the following with your specific values to replace on the whole page and generate copyable commands:

With a platform admin's access key, add tenant.vcluster.com/owner to each cluster-scoped resource the tenant owns, such as its projects, users, and teams. Set its value to the tenant name. Name each one <tenant>--<name>. The prefix is required for every tenant-owned cluster-scoped create, including one a platform admin sends, and a tenant-prefixed name without the matching owner is refused.

project.yaml
apiVersion: management.loft.sh/v1
kind: Project
metadata:
name: acme--team-a
labels:
tenant.vcluster.com/owner: acme
spec:
displayName: Team A

Resources inside a tenant's project take their tenant from the project, so they don't need the label when they're created. The platform adds the label. A later write that replaces the whole object, such as kubectl replace or Argo CD's Replace=true sync option, must keep the same label or it's refused. Client-side and server-side apply keep the live label.

With a tenant user's access key, leave every tenant.vcluster.com/ label out of the manifests. The platform sets the owner label itself and refuses one in a tenant's write. Start each cluster-scoped resource's name with <tenant>--. The platform hides these labels when the tenant reads, so the tool doesn't report them as drift.

With a Kubernetes identity, keep the manifests of resources the tenant already owns as they are. Platform admins and Kubernetes identities can repeat an unchanged owner label when updating or patching an existing resource, but they can't change its tenant. Tenant identities can't send managed labels, even unchanged ones.

Set up a sync for a tenant​

These steps create a tenant's resources with a platform admin's access key and then hand ongoing updates to the tool.

  1. Create a dedicated platform admin user for the GitOps tool, and issue it an access key. See Access keys.

  2. Build a kubeconfig from the access key, as described in Manually create kube config. For Flux, store the kubeconfig in a Secret and set that Secret as the Kustomization's kubeconfig reference.

    For Argo CD, register the management API with a declarative cluster Secret that uses the access key as its bearer token. Don't use argocd cluster add. It replaces the access key with a service account token, and that identity can't create the tenant's resources.

    Modify the following with your specific values to generate a copyable command:
    platform-cluster-secret.yaml
    apiVersion: v1
    kind: Secret
    metadata:
    name: vcluster-platform
    namespace: argocd
    labels:
    argocd.argoproj.io/secret-type: cluster
    type: Opaque
    stringData:
    name: vcluster-platform
    server: https://platform.example.com/kubernetes/management
    config: |
    {
    "bearerToken": "access-key"
    }
  3. Commit the tenant's manifests, written as in Write the manifests, to the repository the tool syncs.

  4. Sync, and confirm the resources were created in the tenant. See Verify.

  5. Optionally, move later syncs to the tool's Kubernetes service account. As long as it has the required RBAC permissions, it can update the tenant's existing projects, users, and teams, but it can't create new ones.

Verify​

  • The tenant's view. Sign in as the tenant admin, or switch to the tenant. Confirm the synced projects are listed. See Verify the boundary.
  • The sync. The Argo CD application reports Synced and Healthy, or the Flux Kustomization reports Ready, with none of the errors above.
  • The permissions. Reapply an unchanged manifest, then sync an intended update with the tool's actual identity. Confirm ownership stays unchanged. In a test environment, confirm that the same identity can't give one of the tenant's teams a role the tenant couldn't grant itself.
  • The ownership. Use a platform admin's access-key kubeconfig to confirm the project includes tenant.vcluster.com/owner with the tenant name. Tenant users' reads hide this label, so their keys can't verify it.
Modify the following with your specific values to generate a copyable command:
kubectl --kubeconfig platform-admin-kubeconfig.yaml get projects.management.loft.sh acme--team-a --show-labels

The LABELS column includes tenant.vcluster.com/owner= followed by the tenant name.

Fix sync errors​

Symptom in the syncCauseFix
a resource's tenant cannot be changedThe manifest added, changed, or removed the owner label on a resource that already exists.Revert the label. To give the resource to another tenant, recreate it there.
RBAC forbidden for the service accountThe sync identity lacks permission for the resource or operation on management.loft.sh.Grant the required RBAC permissions to the sync identity.
is maintained by the platform and cannot be setA tenant identity sent a tenant.vcluster.com/ label, or a Kubernetes identity sent the owner label on a create.Remove the label from a tenant's manifest. Create the resource with a platform admin's access key instead of the Kubernetes identity.
owner tenant does not existThe owner label on a new resource names a tenant that doesn't exist, often a typo.Correct the label to the tenant's name.
must start with "acme--"A tenant-owned cluster-scoped create omitted the prefix, including one sent by a platform admin.Rename it with the prefix.
the object must be owned by itA tenant-prefixed name has no matching owner, for example a Kubernetes identity's create.Create the resource with a platform admin's access key and the matching owner, or use a name outside tenant name prefixes for platform-owned inventory.
not a user or team this tenant can seeA tenant identity's manifest names a user or team that doesn't exist yet or belongs to another tenant. The name is in spec.access, spec.owner, or a team's spec.users.Apply the tenant's users and teams before the resources that name them. See A tenant names only users and teams it can see.
may not own instances of this resourceThe tenant can't create that kind. Its capability is off, the capability doesn't allow the tenant its own, or tenants never author that kind.Create the resource as the platform, or enable allow.custom where the capability offers it.

For more on each refusal, see Troubleshoot tenants.

Next steps​