Skip to main content
Version: v0.30 Stable
Limited vCluster Tenancy Configuration Support

This feature is only available for the following:

Running the control plane as a container and the following worker node types:
Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Sleep mode

Not all workloads need to run continuously. Scaling them down saves time and reduces costs. With sleep mode, you can scale workloads based on a set schedule or user activity and ingress.

warning

Sleep mode is intended for pre-production use cases only, and has limitations when used on a virtual cluster instance not connected to the platform.

Enable sleep mode​

To enable sleep mode, add the following configuration to your vcluster.yaml:

Sleep mode configuration
sleepMode:
  enabled: true
  autoSleep:
    afterInactivity: 1h

Sleep mode operations​

Sleep mode involves two main actions: sleeping and waking. These actions help save resources when the cluster is not in use.

Sleeping reduces resource consumption and costs during inactive periods.

  • Delete bare pods: Pods not managed by controllers are removed permanently.
  • Scale down resources: The following resources are scaled down to reduce resource usage:
    • Deployments
    • ReplicaSets
    • ReplicationControllers
    • DaemonSets

Waking restore scaled-down resources.

  • Restore scaled-down resources: Deployments, ReplicaSets, ReplicationControllers, and DaemonSets are returned to their original state.
  • Bare pods are not restored: Bare pods deleted during the Sleeping step cannot be recovered.

Without an agent​

How it works: The virtual cluster instance manages sleep mode itself by scaling down workloads (pods, Deployments, StatefulSets, etc.) while keeping the control plane active.

What happens:

  • Workloads are scaled down to zero replicas
  • Control plane remains running (API server, etcd, syncer)
  • Cluster can detect activity and wake up automatically
  • Resource savings are moderate (workloads only)

Use when: You want automatic sleep/wake feature but don't have the platform or prefer instance-managed sleep mode.

With a platform agent​

How it works: When connected to the platform using an agent, the platform agent takes over sleep mode management and can shut down the entire control plane.

What happens:

  • Workloads are scaled down to zero replicas
  • Control plane can be completely shut down (maximum resource savings)
  • Platform manages sleep/wake operations
  • Enables additional features like auto-deletion

Use when: You want maximum resource savings and advanced features like auto-deletion, or you're already using the platform for virtual cluster instance management.

Key differences​

FeatureWithout agentWith agent
Control planeRemains activeCan be shut down completely
Resource savingsModerate (workloads only)Maximum (workloads + control plane)
Auto-deletionNot availableAvailable
Managementvirtual cluster instance manages itselfPlatform agent manages
Activity detectionSupportedSupported
SchedulingSupportedSupported

Resource exemption in sleep mode​

In sleep mode, you can prevent specific resources from entering sleep mode by configuring them with the following options:

  • Add the annotation sleepmode.loft.sh/exclude: true to the resource.
  • Configure sleepMode with specific labels to define which resources should remain active. This allows targeting specific resources based on their labels.
  • Add labels to workloads that you want to remain active. By using labels, it is possible to control which resources are exempt from sleep mode at a more granular level.

Detect activity in sleep mode​

Sleep mode tracks certain actions to detect activity and wake the cluster when necessary. To wake a sleeping cluster or update the last active time, sleep mode captures the following actions:

  • Accessing cluster resources through API calls (for example kubectl get <resource>).
  • Attempting to contact ingress endpoints (NGINX and Istio only).
note

Ingress activity detection with the ingress-nginx ingress controller relies on the mirror-target annotation, which will overwrite any previously set mirror-target annotation.

Ignore other types of activity in sleep mode​

The resource exemption feature keeps specifically configured resources active when the virtual cluster instance enters sleep mode. The Configure request handling for sleep mode table describes how to ignore requests that keep the cluster awake or wake it up if it is asleep.

Configure request handling for sleep mode table
AnnotationBehaviorFormat
sleepmode.loft.sh/ignore-allIgnore all activity.true or false
sleepmode.loft.sh/ignore-ingressesIgnore requests to associated ingresses.true or false
sleepmode.loft.sh/ignore-groupsIgnore specified groups the requesting user belongs to.group2,group3
sleepmode.loft.sh/ignore-resourcesIgnore requests to specific resource types.pods,resource2,resource3
sleepmode.loft.sh/ignore-verbsIgnore requests with specific verbs.create,verb1,verb2
sleepmode.loft.sh/ignore-resource-verbsIgnore requests with specific verbs to specific resources.myresource.mygroup=create update delete, myresource2.mygroup=create update
sleepmode.loft.sh/ignore-resource-namesIgnore requests to specific resources with specific names.myresource.mygroup=name1 name2,myotherresource.mygroup=name1
sleepmode.loft.sh/ignore-user-agentsIgnore specific user agents with trailing wildcard support.kubectl*,argo,useragent3
sleepmode.loft.sh/disable-ingress-wakeupIgnore ingress access.true
note

Add the annotations to the virtual cluster instance workload, such as the StatefulSet or Deployment running the virtual cluster. You can dynamically ignore any request by adding the X-Sleep-Mode-Ignore header.

Connect to the platform​

When you connect a virtual cluster instance that has sleep mode configured to the platform (via an agent), the platform agent automatically takes over sleep mode management. The virtual cluster instance stops managing its own sleep mode and the agent handles all sleep/wake operations.

Transition from instance-managed to agent-managed sleep mode​

When a virtual cluster instance with sleep mode configured connects to the platform:

  • The platform reads the existing sleepMode configuration
  • The agent takes over sleep mode management automatically
  • No configuration changes are needed (for vCluster 0.24.0+ and platform 4.3.0+)
  • The virtual cluster instance's sleep mode configuration continues to work, but is now managed by the platform agent

This requires compatible versions of virtual cluster instance and the platform. The vCluster and platform version compatibility with agent installation table provides the steps needed for installing or uninstalling an agent.


vCluster and platform version compatibility with agent installation table
vCluster versionPlatform versionCompatibilityInstalling an agentUninstalling an agent
0.22.x4.2.x✅Update the virtual cluster instance config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
0.23.x4.2.x✅Update the virtual cluster instance config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
0.24.0<4.3.0❌These versions are not compatible, as the vCluster version is ahead of the platform, which causes the virtual cluster instance creation to be rejected.Not applicable.
0.24.0≥4.3.0✅No action is required. The platform reads the unified virtual cluster instance config and takes over as if it had been configured for the platform all along.Remove the annotation vcluster.loft.sh/agent-installed from the virtual cluster instance config secret in the host cluster to notify the virtual cluster instance that it needs to resume sleep mode. The secret’s name is vc-config-[vcluster-name].
≥0.24.0≥4.3.0✅No action is required. The platform reads the unified config and takes over as if it had been configured for the platform all along.No action is required, and the virtual cluster instance resumes sleep mode for workloads only, not the control plane.
vCluster sleep mode considerations
  • In version 0.23.x, the platform does not know how to wake a virtual cluster instance that has put itself to sleep. Ensure the virtual cluster instance is resumed before installing the agent to take over.
  • In version 0.24.x, the autoDelete configuration is still under external.platform and is set to move under sleepMode in a later release. external.platform to be moved under sleepMode in a future release.

Ingress configuration​

Sync to host​

If you install ingress controllers in the virtual cluster instance, prevent them from sleeping. This keeps the controllers active and allows them to handle requests that would wake the virtual cluster instance. If you install the ingress controller in the host cluster, enable ingress syncing to the host.

Enable ingress syncing
sync:
  toHost:
    ingresses:
      enabled: true

Reachability for ingress activity detection​

To enable ingress activity detection in your virtual cluster instance, ensure the following conditions are met:

  • The ingress controller needs to be able to discover the virtual cluster instance pod to route traffic correctly.
  • Ensure that the DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local resolves properly. This allows the ingress controller to reach the virtual cluster instance's service address.
  • Ensure that the correct dnsPolicy is set in the ingress controller's Helm chart. This ensures DNS resolution works as expected and allows communication between the ingress controller and the virtual cluster instance.

Configure sleep mode in a virtual cluster​

The following examples demonstrate how to configure sleep mode in a virtual cluster.

Ensure you install the necessary prerequisites.

Prerequisites​

  • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

    info

    To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

  • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

  • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

  • vCluster: vCluster command-line tool to provision and manage virtual clusters.
brew install loft-sh/tap/vcluster

The binaries in the tap are signed using the Sigstore framework for enhanced security.

Confirm that you've installed the correct version of the vCluster CLI.

vcluster --version
  • docker: Platform for building and running containerized applications
  • kind: Tool for running local Kubernetes clusters in Docker
  • curl: Command-line tool for transferring data over the network

Deployment​

A deployment resource in Kubernetes manages a set of identical pods. Configuring sleep mode for a deployment scales down the pods while keeping the control plane active. This setup allows the virtual cluster to reduce resource usage while still being able to monitor activity and trigger wake-up actions when needed.

Control plane behavior
  • Without agent: The control plane remains active, allowing the virtual cluster instance to monitor activity and wake up automatically.
  • With agent: The platform agent can shut down the control plane completely for maximum resource savings.

This setup allows the virtual cluster to reduce resource usage while still being able to monitor activity and trigger wake-up actions when needed.

Configure sleep mode with a deployment resource

Configure sleep mode for a deployment resource​

  1. Create the kind cluster.

    create kind cluster
    kind create cluster --name sleep-mode-demo

  2. Deploy a virtual cluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s
        exclude:
          selector:
            labels:
              sleep: no-thanks

    And run the following command:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

    Workloads with the label sleep: no-thanks don't enter sleep mode after 30 seconds.

  3. Create demo deployments in your virtual cluster.

    Use the following deployment YAML to create two deployments.

    Example deployments
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sleepy-deployment
      labels:
        app: sleepy-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-1
      template:
        metadata:
          labels:
            app: demo-dep-1
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: sleepy-demo

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: no-sleep-deployment
      labels:
        sleep: no-thanks
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-2
      template:
        metadata:
          labels:
            app: demo-dep-2
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: not-sleepy-demo

    The first deployment does not have any special configurations for sleep mode. You can replace it with another deployment if needed. The second deployment includes a special label on the Deployment, preventing it from scaling down after 30 seconds.

    You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

  4. Verify Deployments sleep status.

    deployment sleep check
    > sleep 30; kubectl get deployments
    NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
    default       no-sleep-deployment   2/2     2            2           1m
    default       sleepy-deployment     0/2     0            0           1m

    The sleepy-deployment reports 0/2 replicas after 30 seconds. Running kubectl counts as cluster activity, which is why it reports 0/2 instead of 0/0. The kubectl command triggers vCluster to update the replica count back to the original value of 2, but the replicas haven't become ready by the time kubectl get ... returns.

Next steps​

Experiment with the sleep mode feature by trying the following:

  • Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
  • Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

Ingress controller​

An ingress controller, such as NGINX, manages external HTTP/S traffic to services in a Kubernetes cluster. To configure sleep mode for the ingress controller, make sure it stays responsive to incoming traffic and can wake up the cluster if needed. This setup keeps the ingress controller active, even when the virtual cluster instance is in sleep mode and allows the controller to handle requests that trigger the virtual cluster to wake up.

Configure sleep mode with an ingress resource

Configure sleep mode for an ingress controller​

  1. Create the kind cluster.

    Create a kind cluster
    kind create cluster --name ingress-demo --config - <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    networking:
      apiServerAddress: "0.0.0.0"
    nodes:
      - role: control-plane
        extraPortMappings:
          - containerPort: 80
            hostPort: 80
            protocol: TCP
    EOF

  2. Install the NGINX IngressController.

    install ingress controller
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.dnsPolicy=ClusterFirstWithHostNet \
      --set controller.hostNetwork=true \
      --set controller.service.type=ClusterIP

  3. Create the vCluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sync:
      toHost:
        ingresses:
          enabled: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s

    And run the following command:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

  4. Enable local DNS resolution for the virtual cluster.

    Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

  5. Create resources for the Ingress such as a Deployment and Service.

    Use the following manifest to create:

    • Namespace named bar
    • Deployment for the pods backing the Service
    • Service for the Ingress
    • Ingress resource
    Example deployments
    apiVersion: v1
    kind: Namespace
    metadata:
      name: bar

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bar-deployment
      namespace: bar
      labels:
        app: bar-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: bar
      template:
        metadata:
          labels:
            app: bar
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: bar-app

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: bar-service
      namespace: bar
    spec:
      selector:
        app: bar
      ports:
        # Default port used by the image
        - port: 8080

    ---

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: example-ingress
      namespace: bar
    spec:
      ingressClassName: nginx
      rules:
        - http:
            paths:
              - pathType: Prefix
                path: /bar
                backend:
                  service:
                    name: bar-service
                    port:
                      number: 8080
          host: backend.local

  6. Verify the ingress is working properly with curl.

    Test the Ingress endpoint within the 30-second activity window by running curl --silent backend.local/bar. The pod name from the Deployment that responds is displayed.

  7. Allow the virtual cluster to go to sleep.

    Wait 30 seconds for the cluster to sleep, then run the curl command again. For convenience, run watch -d curl --silent backend.local/bar to repeatedly test the endpoint. On the first attempt, you’ll see the message Client sent an HTTP request to an HTTPS server. because an HTTP request was sent to the HTTPS wake endpoint. Subsequent requests display new pod names.

Istio gateway​

note

Enable Istio integration for sleep mode to work with Istio resources.

When Istio is installed on the host cluster and the Istio integration is enabled, the virtual cluster instance syncs Gateway and VirtualService resources to the host cluster.

With sleep mode enabled, any traffic routed to a Service in the virtual cluster instance is tracked as activity and keeps the virtual cluster instance awake. If the vCluster is asleep, incoming traffic wakes it up.

Configure an ingress gateway for cluster external access
info

This example continues from the Istio integration example.

  1. Complete the Istio integration example.

  2. Install the Istio ingress gateway controller to allow routing from outside the cluster.

    Save the following manifest as ingress.yaml:

    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    metadata:
      name: ingress
    spec:
      profile: empty # Do not install CRDs or the control plane
      components:
        ingressGateways:
        - name: istio-ingressgateway
          namespace: istio-ingress
          enabled: true
          label:
            istio: ingressgateway
      values:
        gateways:
          istio-ingressgateway:
            injectionTemplate: gateway

  3. Create the istio-ingress namespace and install the ingress gateway:

    kubectl create namespace istio-ingress                                                                                                      
    istioctl install -f ingress.yaml

  4. Create a Gateway resource that uses a selector matching the newly installed ingress gateway in your vCluster.

    apiVersion: networking.istio.io/v1
    kind: Gateway
    metadata:
      name: istio-sm-gateway
      namespace: test
    spec:
      # The selector matches the ingress gateway pod labels.
      # If you installed Istio using Helm following the documentation, this is "istio=ingress"
      selector:
        istio: ingressgateway
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:
        - "smdemo.local"

  5. Update the vCluster with sleepMode enabled.

    Use the following vcluster.yaml to update your virtual cluster on your host cluster with --upgrade. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s
    integrations:
      istio:
        enabled: true

    Run the following command:

    Create vCluster with autoSleep config
    vcluster create <your-vcluster-name> -f vcluster.yaml --upgrade

  6. Enable local DNS resolution for the virtual cluster.

    Add 127.0.0.1 smdemo.local to your /etc/hosts file to match the host configured in the Gateway configuration.

  7. Add the Gateway you created and the smdemo.local host to the VirtualService from the [Istio integration example](../vcluster- yaml/integrations/istio). The updated VirtualService should look like the following:

    Example resources
    apiVersion: networking.istio.io/v1
    kind: VirtualService
    metadata:
      name: nginx-service
      namespace: test
    spec:
      gateways:
        - istio-sm-gateway
      hosts:
        - smdemo.local
      http:
        - match:
            - uri:
                prefix: /v2
          name: nginx-v2
          rewrite:
            uri: /
          route:
            - destination:
                host: nginx-service.test.svc.cluster.local
                subset: v2
        - name: nginx-v1
          route:
            - destination:
                host: nginx-service.test.svc.cluster.local
                subset: v1

  8. Use curl to verify that the Gateway is working correctly.

    Test the Gateway endpoint within the 30-second activity window by running either curl --silent smdemo.local/v1 or curl --silent smdemo.local/v2.

  9. Allow the virtual cluster to go to sleep.

    Wait 30 seconds for the cluster to sleep. Then run the curl command again. To repeatedly test the endpoint, run watch -d curl --silent smdemo.local/v2. While the cluster is asleep or waking up, a 503 error is displayed.

Enable sleep mode with label selectors and schedules​

Use label selectors and schedules to configure sleep mode based on inactivity or specific timing:

Sleep mode label selectors and schedule
Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
sleepMode:
  enabled: true
  autoSleep:
    afterInactivity: 3h  # Uses Go duration with a max unit of hour
    exclude:
      selector:
        labels:
          dont: sleep
Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
sleepMode:
  enabled: true
  timeZone: America/Denver
  autoSleep:
    schedule: 30 17 * * 5
  autoWakeup:
    schedule: 0 7 * * 1

Config reference​

sleepMode required object ​

SleepMode holds the native sleep mode configuration for Pro clusters

enabled required boolean ​

Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

timeZone required string ​

Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

autoSleep required object ​

AutoSleep holds autoSleep details

afterInactivity required string ​

AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

schedule required string ​

Schedule represents a cron schedule for when to sleep workloads

exclude required object ​

Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

selector required object ​
labels required object ​

Labels defines what labels should be looked for

autoWakeup required object ​

AutoWakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

schedule required string ​