Skip to main content
Version: v0.27 Stable

Private Nodes

Limited vCluster Tenancy Configuration Support

This feature is only available when using the following worker node types:

  • Private Nodes
    • Enterprise-Only Feature

    This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

    Using private nodes is a tenancy model for vCluster where, instead of sharing the host cluster’s worker nodes, you assign individual worker nodes to a vCluster instance. These private nodes act as the vCluster’s worker nodes and are treated as worker nodes for the vCluster.

    Because these nodes are real Kubernetes nodes, vCluster does not sync any resources to the host cluster as no host cluster worker nodes are used. All workloads run directly on the attached nodes as if they were native to the virtual cluster.

    This approach can be understood as follows: the control plane creates a virtual Kubernetes context, while the workloads can run on separate physical nodes that exist entirely within that virtual context.

    One key benefit of using private nodes is that vCluster can automatically manage the lifecycle of the worker nodes.

    To allow nodes to join the virtual cluster, the cluster must be exposed and accessible. One option is to expose a LoadBalancer service and use its endpoint for the worker nodes. vCluster does not need direct access to the nodes; instead, it uses Konnectivity to tunnel requests—such as kubectl logs—from the control plane to the nodes.


    Overview

    vcluster.yaml configuration​

    To create a vCluster with private nodes, you'll need to enable the feature as well as set some other configuration options in preparation for worker nodes to join.

    Enable private nodes
    # Enable private nodes
    privateNodes:
      enabled: true

    # vCluster control plane options
    controlPlane:
      distro:
        k8s:
          image:
            tag: v1.31.2 # Kubernetes version you want to use
      service:
        spec:
          type: LoadBalancer # If you want to expose vCluster using LoadBalancer (Recommended option)

    # Networking configuration
    networking:
      # Specify the pod CIDR
      podCIDR: 10.64.0.0/16
      # Specify the service CIDR
      serviceCIDR: 10.128.0.0/16

    Scheduling enabled by default​

    When private nodes are enabled, scheduling is handled by the virtual cluster and the virtual scheduler is automatically enabled. You do not need to manually configure this in your vcluster.yaml.

    Virtual scheduler automatically enabled with private nodes
    # Enabling private nodes enables the virtual scheduler
    # Disabling the virtual scheduler is not possible
    controlPlane:
      advanced:
        virtualScheduler:
          enabled: true

    Other vCluster feature limitations​

    When private nodes are enabled, certain vCluster features are automatically disabled or unavailable. If you include these options in your vcluster.yaml, they are ignored or might cause configuration errors.

    The following features are not available when using private nodes:

    • sync.* - No resource syncing between virtual and host clusters
    • integrations.* - Integrations depend on syncing functionality
    • networking.replicateServices - Services are not replicated to host
    • controlPlane.distro.k3s - Only standard Kubernetes (k8s) is supported
    • controlPlane.coredns.embedded: true - Embedded CoreDNS conflicts with custom CNI
    • controlPlane.advanced.virtualScheduler.enabled: false - Virtual scheduler cannot be disabled
    • sleepMode.* - No ability to sleep workloads or control plane
    Unsupported vcluster.yaml options with private nodes
    # These configurations are NOT supported with private nodes

    # Resource syncing between virtual and host clusters is disabled
    sync:
      services:
        enabled: false  # Services cannot be synced to host cluster
      secrets:
        enabled: false  # Secrets cannot be synced to host cluster
                        # All other sync options (pods, configmaps, etc.) are also disabled

    # Platform integrations require syncing functionality
    integrations:
      metricsServer:
        enabled: false  # Metrics server integration not supported
                        # All other integrations are disabled due to sync dependency

    # Service replication to host cluster is not available
    networking:
      replicateServices:
        enabled: false  # Services run entirely within virtual cluster

    # Distribution restrictions
    controlPlane:
      distro:
        k3s:
          enabled: false  # k3s distribution not supported
        k8s:
          enabled: true   # Only standard Kubernetes works

      # DNS configuration limitations
      coredns:
        embedded: false   # Embedded CoreDNS conflicts with custom CNI options
      advanced:
        # Virtual scheduler is required for workload placement
        virtualScheduler:
          enabled: true   # Always enabled (cannot be disabled)
      # Host Path Mapper is not supported
      hostPathMapper:
        enabled: false

    # Sleep mode is not available
    sleepMode:
      enabled: false

    # Isolated Control Plane mode is not supported
    experimental:
      isolatedControlPlane:
        enabled: false

    Configure addons​

    vCluster supports addons that extend the functionality of your virtual cluster. You can configure these addons during deployment to adjust networking, observability, and other features for your environment and requirements.

    CNI​

    vCluster installs Flannel by default. You can install your own CNI with the following.

    Disable default Flannel CNI
    privateNodes:
      cni:
        flannel:
          enabled: false

    CoreDNS​

    vCluster installs CoreDNS by default to allow cluster DNS resolving of services and pods.

    Konnectivity​

    vCluster uses Konnectivity to establish a connection between the control plane and worker nodes for commands such as kubectl logs, kubectl exec, kubectl port-forward. The Konnectivity server runs as part of the vCluster control plane, while an agent is deployed inside the vCluster. If it is not needed or wanted, you can disable Konnectivity in the vCluster config.

    Kube proxy​

    vCluster installs Kube Proxy by default to ensure that services are configured on the nodes. Some CNI implement their own kube-proxy functionality. Optionally, you can disable kube-proxy deployment in the vCluster config.

    Local Path Provisioner​

    vCluster installs the Local Path Provisioner by default to allow stateful workloads within the vCluster.

    Config reference​

    privateNodes required object ​

    PrivateNodes holds configuration for vCluster private nodes mode.

    enabled required boolean false ​

    Enabled defines if dedicated nodes should be enabled.

    kubelet required object ​

    Kubelet holds kubelet configuration that is used for all nodes.

    config required object {} ​

    Config is the config for the kubelet that will be merged into the default kubelet config. More information can be found here: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration

    autoUpgrade required object ​

    AutoUpgrade holds configuration for auto upgrade.

    enabled required boolean true ​

    Enabled defines if auto upgrade should be enabled.

    image required string ​

    Image is the image for the auto upgrade pod started by vCluster. If empty defaults to the controlPlane.statefulSet.image.

    imagePullPolicy required string ​

    ImagePullPolicy is the policy how to pull the image.

    nodeSelector required object ​

    NodeSelector is the node selector for the auto upgrade. If empty will select all worker nodes.

    binariesPath required string ​

    BinariesPath is the base path for the kubeadm binaries. Defaults to /usr/local/bin

    cniBinariesPath required string ​

    CNIBinariesPath is the base path for the CNI binaries. Defaults to /opt/cni/bin

    concurrency required integer 1 ​

    Concurrency is the number of nodes that can be upgraded at the same time.

    joinNode required object ​

    JoinNode holds configuration specifically used during joining the node (see "kubeadm join").

    preInstallCommands required string[] ​

    PreInstallCommands are commands that will be executed before containerd, kubelet etc. is installed.

    preJoinCommands required string[] ​

    PreJoinCommands are commands that will be executed before kubeadm join is executed.

    postJoinCommands required string[] ​

    PostJoinCommands are commands that will be executed after kubeadm join is executed.

    containerd required object ​

    Containerd holds configuration for the containerd join process.

    enabled required boolean true ​

    Enabled defines if containerd should be installed and configured by vCluster.

    registry required object ​

    Registry holds configuration for how containerd should be configured to use a registries.

    configPath required string ​

    ConfigPath is the path to the containerd registry config.

    mirrors required {key: object} ​

    Mirrors holds configuration for the containerd registry mirrors. E.g. myregistry.io:5000 or docker.io. See https://github.com/containerd/containerd/blob/main/docs/hosts.md for more details.

    server required string ​

    Server is the fallback server to use for the containerd registry mirror. E.g. https://registry-1.docker.io. See https://github.com/containerd/containerd/blob/main/docs/hosts.md for more details.

    caCert required string[] ​

    CACert are paths to CA certificates to use for the containerd registry mirror.

    skipVerify required boolean ​

    SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections.

    capabilities required string[] ​

    Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities.

    hosts required object[] ​

    Hosts holds configuration for the containerd registry mirror hosts. See https://github.com/containerd/containerd/blob/main/docs/hosts.md for more details.

    server required string ​

    Server is the server to use for the containerd registry mirror host. E.g. http://192.168.31.250:5000.

    caCert required string[] ​

    CACert are paths to CA certificates to use for the containerd registry mirror host.

    skipVerify required boolean ​

    SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections.

    capabilities required string[] ​

    Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities.

    pauseImage required string ​

    PauseImage is the image for the pause container.

    caCertPath required string ​

    CACertPath is the path to the SSL certificate authority used to secure communications between node and control-plane. Defaults to "/etc/kubernetes/pki/ca.crt".

    skipPhases required string[] ​

    SkipPhases is a list of phases to skip during command execution. The list of phases can be obtained with the "kubeadm join --help" command.

    nodeRegistration required object ​

    NodeRegistration holds configuration for the node registration similar to the kubeadm node registration.

    criSocket required string ​

    CRI socket is the socket for the CRI.

    kubeletExtraArgs required object[] ​

    KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config ConfigMap Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. An argument name in this list is the flag name as it appears on the command line except without leading dash(es). Extra arguments will override existing default arguments. Duplicate extra arguments are allowed.

    name required string ​

    Name is the name of the argument.

    value required string ​

    Value is the value of the argument.

    taints required object[] ​

    Taints are additional taints to set for the kubelet.

    key required string ​

    Required. The taint key to be applied to a node.

    value required string ​

    The taint value corresponding to the taint key.

    effect required string ​

    Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.

    ignorePreflightErrors required string[] ​

    IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered, e.g. 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.

    imagePullPolicy required string ​

    ImagePullPolicy specifies the policy for image pulling during kubeadm "init" and "join" operations. The value of this field must be one of "Always", "IfNotPresent" or "Never". If this field is unset kubeadm will default it to "IfNotPresent", or pull the required images if not present on the host.

    nodePools required object ​

    NodePools stores karpenter node pool configuration

    static required object[] ​

    Static defines static node pools. Static node pools have a fixed size and are not scaled automatically.

    name required string ​

    Name is the name of this static nodePool

    requirements required object[] ​

    Requirements filter the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible.

    property required string ​

    Property is the property on the node type to select.

    operator required string ​

    Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In".

    values required string[] ​

    Values is the list of values to use for comparison. This is mutually exclusive with value.

    value required string ​

    Value is the value to use for comparison. This is mutually exclusive with values.

    taints required object[] ​

    Taints are the taints to apply to the nodes in this pool.

    key required string ​

    Required. The taint key to be applied to a node.

    value required string ​

    The taint value corresponding to the taint key.

    effect required string ​

    Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.

    nodeLabels required object ​

    NodeLabels are the labels to apply to the nodes in this pool.

    terminationGracePeriod required string ​

    TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.

    Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.

    This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.

    Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.

    The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. Defaults to 30s. Set to Never to wait indefinitely for pods to be drained.

    quantity required integer ​

    Quantity is the number of desired nodes in this pool.

    dynamic required object[] ​

    Dynamic defines dynamic node pools. Dynamic node pools are scaled automatically based on the requirements within the cluster. Karpenter is used under the hood to handle the scheduling of the nodes.

    name required string ​

    Name is the name of this NodePool

    requirements required object[] ​

    Requirements filter the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible.

    property required string ​

    Property is the property on the node type to select.

    operator required string ​

    Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In".

    values required string[] ​

    Values is the list of values to use for comparison. This is mutually exclusive with value.

    value required string ​

    Value is the value to use for comparison. This is mutually exclusive with values.

    taints required object[] ​

    Taints are the taints to apply to the nodes in this pool.

    key required string ​

    Required. The taint key to be applied to a node.

    value required string ​

    The taint value corresponding to the taint key.

    effect required string ​

    Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.

    nodeLabels required object ​

    NodeLabels are the labels to apply to the nodes in this pool.

    limits required object ​

    Limits specify the maximum resources that can be provisioned by this node pool, mapping to the 'limits' field in Karpenter's NodePool API.

    disruption required object ​

    Disruption contains the parameters that relate to Karpenter's disruption logic

    consolidateAfter required string ​

    ConsolidateAfter is the duration the controller will wait before attempting to terminate nodes that are underutilized. Refer to ConsolidationPolicy for how underutilization is considered.

    consolidationPolicy required string ​

    ConsolidationPolicy describes which nodes Karpenter can disrupt through its consolidation algorithm. This policy defaults to "WhenEmptyOrUnderutilized" if not specified

    budgets required object[] ​

    Budgets is a list of Budgets. If there are multiple active budgets, Karpenter uses the most restrictive value. If left undefined, this will default to one budget with a value to 10%.

    nodes required string ​

    Nodes dictates the maximum number of NodeClaims owned by this NodePool that can be terminating at once. This is calculated by counting nodes that have a deletion timestamp set, or are actively being deleted by Karpenter. This field is required when specifying a budget.

    schedule required string ​

    Schedule specifies when a budget begins being active, following the upstream cronjob syntax. If omitted, the budget is always active. Timezones are not supported.

    duration required string ​

    Duration determines how long a Budget is active since each Schedule hit. Only minutes and hours are accepted, as cron does not work in seconds. If omitted, the budget is always active. This is required if Schedule is set.

    terminationGracePeriod required string ​

    TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.

    Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.

    This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.

    Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.

    The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. Defaults to 30s. Set to Never to wait indefinitely for pods to be drained.

    expireAfter required string ​

    The amount of time a Node can live on the cluster before being removed

    weight required integer ​

    Weight is the weight of this node pool.

    tunnel required object ​

    Tunnel holds configuration for the private nodes tunnel. This can be used to connect the private nodes to the control plane or connect the private nodes to each other if they are not running in the same network. Platform connection is required for the tunnel to work.

    enabled required boolean false ​

    Enabled defines if the private nodes tunnel should be enabled.

    nodeToNode required object ​

    NodeToNode holds configuration for the node to node tunnel. This can be used to connect the private nodes to each other if they are not running in the same network.

    enabled required boolean false ​

    Enabled defines if the node to node tunnel should be enabled.