Skip to main content
Version: main 🚧

Clean obsolete nodes after changing the node selector

Issue​

After configuring a node selector in vCluster that previously had node syncing enabled without selectors, outdated nodes remain visible in the cluster.

You might encounter this issue if you observe the following after configuring a node selector in vCluster:

  • Nodes that no longer match your selector criteria still appear in vCluster.
  • These nodes persist even after vCluster restarts.
  • You see more nodes than expected when running kubectl get nodes.
  • Node count doesn't decrease after changing the node selector configuration.
  • Events and logs don't show any errors related to node syncing.
  • The output of kubectl get nodes shows nodes that shouldn't be included based on your current node selector settings.
  • Pods might still be scheduled to these obsolete nodes until they're manually removed.
  • Node labels show the original state before your node selector configuration was updated.
  • Resource utilization metrics continue to be reported for nodes that should no longer be visible.

Cause​

When node syncing is initially enabled without a specified node selector, vCluster syncs nodes based on where pods are scheduled. If you later configure a node selector, previously synced nodes aren't automatically removed when vCluster restarts.

Solution​

  1. Switch to the vCluster context.

    Ensure you're connected to your vCluster:

    vcluster connect my-vcluster -n my-namespace

  2. Identify nodes to remove.

    List all nodes in your vCluster:

    kubectl get nodes

    Identify which nodes no longer match your current node selector configuration.

  3. Delete the obsolete nodes.

    Remove each unnecessary node using the following command:

    kubectl delete node <node-name>

    If any pods are running on these nodes, they are rescheduled to the remaining nodes.

Prevention​

To avoid this issue in the future:

  • Configure node selectors before enabling node syncing if possible.
  • Document your node selector configuration to avoid unexpected changes.
  • After changing node selectors, check and clean up any remaining nodes.

Additional resources​

For more information about configuring node syncing in vCluster, see to the node syncing documentation.