Skip to main content
Version: main 🚧

Hybrid Scheduling

Hybrid scheduling enables virtual clusters to use multiple schedulers simultaneously - both from the host cluster and virtual cluster. Pods within the virtual cluster can use different schedulers based on their individual configuration.

You can enable hybrid scheduling using the following:

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true
        hostSchedulers:
        - my-custom-scheduler
  fromHost:
    nodes:
      enabled: true

You must also enable syncing of real nodes from the host to the virtual cluster. If you don’t, the virtual cluster displays an error.

By default, all nodes from the host cluster are synced to the virtual cluster. To sync only specific nodes, you can define a label selector as shown in the following example:

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true
        hostSchedulers:
        - my-custom-scheduler
  fromHost:
    nodes:
      enabled: true
      labels:
        environment: production
        team: backend

Use host schedulers​

sync.toHost.pods.hybridScheduling.hostSchedulers specifies a list of schedulers available on the host cluster that virtual cluster pods can use.

The default Kubernetes scheduler on the host cluster is always implicitly included in the sync.toHost.pods.hybridScheduling.hostSchedulers list.

A default scheduler can be used in two ways: by omitting .spec.schedulerName or by explicitly setting it to default-scheduler. In both cases, the default Kubernetes scheduler running on the host cluster schedules the pod.

Use custom host schedulers​

The following hybrid scheduling configuration allows virtual cluster pods to use the host cluster's custom schedulers my-custom-scheduler and my-gpu-scheduler.

note

This example omits the full vCluster configuration to highlight the relevant hybrid scheduling settings.

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true
        hostSchedulers:
        - my-custom-scheduler
        - my-gpu-scheduler

You can now deploy pods in the virtual cluster that use the host cluster’s custom schedulers my-custom-scheduler and my-gpu-scheduler. For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-example
spec:
  schedulerName: my-custom-scheduler
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10
---
apiVersion: v1
kind: Pod
metadata:
  name: my-gpu-example
spec:
  schedulerName: my-gpu-scheduler
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10

Use custom virtual schedulers​

Virtual cluster pods can use a custom scheduler that runs inside the virtual cluster.
To ensure these pods are scheduled correctly, do not include virtual schedulers in the sync.toHost.pods.hybridScheduling.hostSchedulers list in your vCluster configuration.

The following example shows the hybrid scheduling configuration and a pod that uses the my-virtual-ai-scheduler running in the virtual cluster.

Hybrid scheduling configuration:

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true

Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-ai-example
spec:
  schedulerName: my-virtual-ai-scheduler
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10

When hybrid scheduling is enabled and PersistentVolumeClaims are synced from the virtual cluster to the host cluster, additional resources must also be present in the virtual cluster for scheduling to function correctly.

By default, vCluster automatically syncs the following resources from the host to the virtual cluster, unless this behavior is explicitly disabled in the configuration:

  • CSINodes,
  • CSIStorageCapacities
  • CSIDrivers

When PersistentVolumeClaims are synced to the host cluster, vCluster also enables syncing of StorageClasses from the host to the virtual cluster. However, this only happens if syncing StorageClasses from the virtual cluster to the host is disabled in the configuration.

warning

Manually disabling CSINodes, CSIStorageCapacities, CSIDrivers and StorageClasses sync in the above cases can cause pod scheduling to fail in your virtual cluster.

vCluster automatically adds the required ClusterRole rules to ensure syncing works correctly for automatically enabled CSINodes, CSIStorageCapacities, CSIDrivers, and StorageClasses.