Skip to main content
Version: main 🚧

Rotate certificates

vCluster client and server certificates (also called leaf certificates) remain valid for 1 year from the initial virtual cluster creation. By default, vCluster uses a self-signed Certificate Authority (CA) to sign these certificates. The CA certificate remains valid for 10 years.

vCluster provides commands that allow you to check details about the certificates as well as rotate them.

info

Checking and rotating certificates is only supported for virtual clusters using the k8s distribution.

important

The virtual cluster Pod must be in a running state otherwise the cert commands below abort the operation. In case of certificate expiration during runtime of a virtual cluster it will still be in a running state (though, non-functional) and the rotation can still be performed.

Check certificate information​

The vcluster certs check command provides the following information for each certificate of the given virtual cluster:


FieldDescription
FilenameThe name of the certificate file.
SubjectThe common name of the subject of the certificate.
IssuerThe common name of the issuer of the certificate.
Expires OnThe date when the certificate expires.
StatusThe expiry status of the certificate (either OK, EXPIRED or NOT YET VALID).

The following command shows this information as a table. You can change the format by using --output:


Modify the following with your specific values to generate a copyable command:
vcluster -n my-vcluster certs check my-vcluster

Example output:

            FILENAME           |            SUBJECT            |     ISSUER     |       EXPIRES ON       | STATUS
-------------------------------+-------------------------------+----------------+------------------------+---------
  apiserver-etcd-client.crt    | kube-apiserver-etcd-client    | etcd-ca        | Jul 14, 2026 19:24 UTC | OK
  apiserver-kubelet-client.crt | kube-apiserver-kubelet-client | kubernetes     | Jul 14, 2026 19:24 UTC | OK
  apiserver.crt                | kube-apiserver                | kubernetes     | Jul 14, 2026 19:24 UTC | OK
  ca.crt                       | kubernetes                    | kubernetes     | Jul 12, 2035 19:24 UTC | OK
  client-ca.crt                | kubernetes                    | kubernetes     | Jul 12, 2035 19:24 UTC | OK
  ca.crt                       | etcd-ca                       | etcd-ca        | Jul 12, 2035 19:24 UTC | OK
  healthcheck-client.crt       | kube-etcd-healthcheck-client  | etcd-ca        | Jul 14, 2026 19:24 UTC | OK
  peer.crt                     | vcluster                      | etcd-ca        | Jul 14, 2026 19:24 UTC | OK
  server.crt                   | vcluster                      | etcd-ca        | Jul 14, 2026 19:24 UTC | OK
  front-proxy-ca.crt           | front-proxy-ca                | front-proxy-ca | Jul 12, 2035 19:24 UTC | OK
  front-proxy-client.crt       | front-proxy-client            | front-proxy-ca | Jul 14, 2026 19:24 UTC | OK
  server-ca.crt                | kubernetes                    | kubernetes     | Jul 12, 2035 19:24 UTC | OK

Rotate client and server leaf certificates​

The certificate rotation command performs the following operations:

  • Creates PKI backup: Backs up the existing Public Key Infrastructure (PKI) directory to preserve the current certificate state.
  • Generates new certificates: Creates new client and server leaf certificates signed by the existing Certificate Authority.
  • Restarts virtual cluster: Restarts the virtual cluster to load the new certificates.

Control-plane components (API server, etcd, kubelet) require the restart because they cache certificates in memory during startup. Without a restart, these components continue using the expired or soon-to-expire certificates, which causes authentication failures.

The restart does not affect running application workloads because the underlying cluster nodes and container runtime remain operational during the virtual cluster restart process.


Modify the following with your specific values to generate a copyable command:
vcluster -n my-vcluster certs rotate my-vcluster

Client and server leaf certificate rotation is less intrusive compared to CA rotation. This means that connecting to the virtual cluster with previously issued kubeconfigs continues to work.

Changing the CA certificate​

warning

CA rotation replaces the Certificate Authority that validates all certificates in the cluster. This breaks existing client connections and requires regenerating all kubeconfigs. Use this command carefully, as it affects all cluster access.

When working with virtual clusters with private nodes, there are extra steps that need to happen before doing any changes to the CA certificate and after making the changes to the CA certificate.

Rotate the CA Certificate​

The CA rotation command performs these operations:

  • Creates PKI backup: Backs up the existing certificate directory to enable recovery if needed.
  • Generates new Certificate Authority: Creates a new CA certificate and uses it to sign fresh certificates for all cluster components.
  • Restarts virtual cluster: Restarts the cluster to load the new certificates.

If the rotation fails, you can restore the previous PKI from the automatically created backup. For more information, see the documentation on restoring the previous CA from a backup for recovery.


Modify the following with your specific values to generate a copyable command:
vcluster -n my-vcluster certs rotate-ca my-vcluster

As this replaces the trust anchor completely, everything configured to trust the old CA becomes invalidated (for instance previously issued kubeconfigs).

Restore the previous CA from a backup​

The rotation creates a backup in the virtual cluster's data directory using the following pattern:

/data/pki.bak/<unixtimestamp>
warning

Use caution when performing the following operations. Verify each command before execution.

  1. To restore from the backup, access the virtual cluster syncer container:

    Modify the following with your specific values to generate a copyable command:
    kubectl -n my-vcluster exec -ti svc/my-vcluster -c syncer -- sh

  2. List the backups and choose the latest:

    ls -lat /data/pki.bak/

  3. Delete the PKI directory and copy the backup to that location:

    Modify the following with your specific values to generate a copyable command:
    rm -rf /data/pki
    cp -R /data/pki.bak/1752736900 /data/pki

  4. Log out and perform a regular leaf certificate rotation:

    Modify the following with your specific values to generate a copyable command:
    vcluster -n my-vcluster certs rotate my-vcluster

Additional steps required for private nodes​

There are additional steps required for private nodes. Depending on how your private worker nodes were added will indicate what needs to be done.

There are two types of private work nodes:

  • Manually Provisioned Nodes - These nodes are manually provisioned and joined to the vCluster. The user needs to provision and join these nodes.
  • Automatically Provisioned Nodes - These nodes are automatically provisioned and joined to the vCluster by the vCluster and vCluster Platform.

Manually provisioned private nodes​

Manually provisioned nodes that are managed outside of the vCluster require additional steps before and after rotating the CA certificate.

Before changing the CA certificate​

important

Node deletion needs to happen before any CA cert changes whether it's rotating or restoring the cert. Otherwise, the kubelet running on the node is not able to communicate with the apiserver running with the updated certificate.

You must delete all previously joined manually provisioned private nodes. After the rotation, you must re-join the nodes again to the virtual cluster. Each step must be performed on each manually provisioned private node.

  1. Connect to your virtual cluster:

    Modify the following with your specific values to generate a copyable command:
    vcluster -n my-vcluster connect my-vcluster

  2. Delete each manually provisioned private node:

    info

    vcluster node delete tries to evict any workload before deleting the node from the virtual cluster. If there's no other node for the workload to be re-scheduled to, the workload stops. If you forgot to delete a node before change the CA cert, you must append --drain=false to successfully delete the node.

    Modify the following with your specific values to generate a copyable command:
    vcluster node delete my-node

After changing the CA certificate​

After a CA rotation or restoration, you need to re-create a token to join the virtual cluster and join each manually provisioned private node back to the virtual cluster.

  1. Connect to your virtual cluster:

    Modify the following with your specific values to generate a copyable command:
    vcluster -n my-vcluster connect my-vcluster

  2. Create a new join token:

    vcluster token create

  3. Copy the command and append the --force-join flag. Run the updated command on every private node.

    warning

    The --force-join flag leads to running containers being killed via the container runtime and being restarted by the kubelet. If you have a stateful application running that requires graceful termination, please ensure that you have performed the necessary steps beforehand (vcluster node delete, backups etc.).

    curl -fsSLk "https://example.com:443/node/join?token=d34db33f" | sh -s -- --force-join

Automatically provisioned private nodes​

Automatically provisioned nodes that are managed by the vCluster only require additional steps after rotating the CA certificate.

After changing the CA certificate​

All logic of the nodes that need to be provisioned is stored in vCluster Platform. After a CA rotation or restoration, you need to delete all NodeClaims related to the virtual cluster, but those NodeClaims are in the Kubernetes cluster that has vCluster Platform deployed on.

  1. Connect to the Kubernetes cluster that is hosting your vCluster Platform so you can run kubectl commands.

  2. List current NodeClaims in vCluster Platform.

    kubectl get nodeclaims.management.loft.sh -A

    NAMESPACE   NAME             STATUS      VCLUSTER     NODETYPE   CREATED AT
    p-default   vcluster-4zbg8   Available   vcluster     medium     2025-08-28T11:08:18Z
    p-default   vcluster-5xtx3   Available   vcluster-2   medium     2025-08-27T10:01:33Z

  3. Find the relevant NodeClaims for the virtual cluster.

    From the list of NodeClaims, identify the ones associated to your virtual cluster that had the CA certificate change.

    In this example, if you have rotated the CA certificate of the virtual cluster named vcluster-2, then the associated NodeClaim is vcluster-5xtx3.

  4. Delete the NodeClaim of the virtual cluster that had a CA change.

    For the virtual cluster that you had changed the CA certificate, you'll need to delete each NodeClaim, which removes the NodeClaim and deletes the underlying Node. vCluster Platform recreates the NodeClaim and the new nodes re-join the virtual cluster using the new CA certificate.

    Example of deleting the NodeClaim for a single virtual cluster
    kubectl -n p-default delete nodeclaims.management.loft.sh vcluster-5xtx3 

    nodeclaim.management.loft.sh "vcluster-5xtx3" deleted