Skip to main content
Version: v0.38 Stable

Resolve embedded etcd defragmentation timeouts on startup

Symptoms​

The control plane doesn't start. Its logs contain this line:

Etcd defragmentation did not finish in time, increase controlPlane.backingStore.etcd.embedded.defragTimeout

The startup error that follows contains defragment: and a deadline error from etcd.

Cause​

Embedded etcd defragments its database each time it starts, when at least 16 MiB and 25% of the database are reclaimable. The control plane waits for defragmentation to finish before it serves requests. If it takes longer than controlPlane.backingStore.etcd.embedded.defragTimeout, which defaults to 5m, the control plane fails to start.

This happens with large databases, slow storage, or both. It applies only to clusters that use embedded etcd.

Fix​

Raise the timeout, or skip defragmentation on startup. The control plane reads the setting at startup, so applying the change restarts it.

  1. Set a longer timeout in your vcluster.yaml. To skip defragmentation instead, set defragTimeout: 0s.

    vcluster.yaml
    controlPlane:
    backingStore:
    etcd:
    embedded:
    enabled: true
    defragTimeout: 15m
  2. Apply the change with the same method you used to deploy the cluster, such as the CLI or a Helm upgrade.

If you skip defragmentation, the database file doesn't shrink after compaction. Watch its size, because a database that reaches its quota raises the NOSPACE alarm.

Verify​

The control plane pods reach Running, and the logs show one of these lines instead of the timeout:

Defragmenting etcd database
Skipping etcd defragmentation, not enough reclaimable space
Skipping etcd defragmentation, disabled via controlPlane.backingStore.etcd.embedded.defragTimeout