Skip to main content
Version: main 🚧
Limited vCluster Tenancy Configuration Support

This feature is only available when using the following worker node types:

  • Host Nodes
    • Enterprise-Only Feature

    This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

    Sleep mode

    Not all workloads need to run continuously. Scaling them down saves time and reduces costs. With sleep mode, you can scale workloads based on a set schedule or user activity and ingress.

    warning

    Sleep mode is intended for pre-production use cases only, and has limitations when used on a standalone vCluster not connected to the platform.

    Enable sleep mode​

    To enable sleep mode, add the following configuration to your vcluster.yaml:

    Sleep mode configuration
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 1h

    Sleep mode operations​

    Sleep mode involves two main actions: sleeping and waking. These actions help save resources when the cluster is not in use.

    Sleeping reduces resource consumption and costs during inactive periods.

    • Delete bare pods: Pods not managed by controllers are removed permanently.
    • Scale down resources: The following resources are scaled down to reduce resource usage:
      • Deployments
      • ReplicaSets
      • ReplicationControllers
      • DaemonSets

    Waking restore scaled-down resources.

    • Restore scaled-down resources: Deployments, ReplicaSets, ReplicationControllers, and DaemonSets are returned to their original state.
    • Bare pods are not restored: Bare pods deleted during the Sleeping step cannot be recovered.

    Resource exemption in sleep mode​

    In sleep mode, you can prevent specific resources from entering sleep mode by configuring them with the following options:

    • Add the annotation sleepmode.loft.sh/exclude: true to the resource.
    • Configure sleepMode with specific labels to define which resources should remain active. This allows targeting specific resources based on their labels.
    • Add labels to workloads that you want to remain active. By using labels, it is possible to control which resources are exempt from sleep mode at a more granular level.

    Detect activity in sleep mode​

    Sleep mode tracks certain actions to detect activity and wake the cluster when necessary. To wake a sleeping cluster or update the last active time, sleep mode captures the following actions:

    • Accessing cluster resources through API calls (for example kubectl get <resource>).
    • Attempting to contact ingress endpoints (NGINX only).
    note

    Ingress activity detection works only with NGINX ingress controllers, using the mirror-target annotation. This overwrites any previously set mirror-target annotation.

    Ignore other types of activity in sleep mode​

    The resource exemption feature keeps specifically configured resources active when the vCluster enters sleep mode. The Configure request handling for sleep mode table describes how to ignore requests that keep the cluster awake or wake it up if it is asleep.

    Configure request handling for sleep mode table
    AnnotationBehaviorFormat
    sleepmode.loft.sh/ignore-allIgnore all activity.true or false
    sleepmode.loft.sh/ignore-ingressesIgnore requests to associated ingresses.true or false
    sleepmode.loft.sh/ignore-groupsIgnore specified groups the requesting user belongs to.group2,group3
    sleepmode.loft.sh/ignore-resourcesIgnore requests to specific resource types.pods,resource2,resource3
    sleepmode.loft.sh/ignore-verbsIgnore requests with specific verbs.create,verb1,verb2
    sleepmode.loft.sh/ignore-resource-verbsIgnore requests with specific verbs to specific resources.myresource.mygroup=create update delete, myresource2.mygroup=create update
    sleepmode.loft.sh/ignore-resource-namesIgnore requests to specific resources with specific names.myresource.mygroup=name1 name2,myotherresource.mygroup=name1
    sleepmode.loft.sh/ignore-user-agentsIgnore specific user agents with trailing wildcard support.kubectl*,argo,useragent3
    sleepmode.loft.sh/disable-ingress-wakeupIgnore ingress access.true
    note

    Add the annotations to the vCluster workload, such as the StatefulSet or Deployment running the virtual cluster. You can dynamically ignore any request by adding the X-Sleep-Mode-Ignore header.

    vCluster sleep mode compatibility with the platform​

    You can configure sleep mode in a virtual cluster in two ways:

    • Without an agent: This scales down workloads (like pods) while keeping the control plane active. The cluster remains in a minimal state and can wake up when activity is detected.

    • With an agent: When connected to the platform using an agent, the agent fully manages sleep mode by shutting down the control plane. This reduces resource usage more and enables features like auto-deletion, which aren’t available without the agent.

    When a vCluster connects to the platform using an agent, the agent manages sleep mode actions by shutting down the control plane completely. This reduces resource usage and enables features like auto-deletion, which is not available when the vCluster is not connected to the platform using an agent.

    Connect to the platform​

    When you connect a vCluster configured for sleep mode to the platform, it stops managing its own sleep mode actions and allows the agent to take control. This requires compatible versions of vCluster and the platform. The vCluster and platform version compatibility with agent installation table provides the steps needed for installing or uninstalling an agent.


    vCluster and platform version compatibility with agent installation table
    vCluster versionPlatform versionCompatibilityInstalling an agentUninstalling an agent
    0.22.x4.2.x✅Update the vCluster config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
    0.23.x4.2.x✅Update the vCluster config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
    0.24.0<4.3.0❌These versions are not compatible, as the vCluster version is ahead of the platform, which causes the vCluster creation to be rejected.Not applicable.
    0.24.0≥4.3.0✅No action is required. The platform reads the unified vCluster config and takes over as if it had been configured for the platform all along.Remove the annotation vcluster.loft.sh/agent-installed from the vCluster config secret in the host cluster to notify the vCluster that it needs to resume sleep mode. The secret’s name is vc-config-[vcluster-name].
    ≥0.24.0 (future release)≥4.3.0✅No action is required. The platform reads the unified config and takes over as if it had been configured for the platform all along.No action is required, and the vCluster resumes sleep mode for workloads only, not the control plane.
    vCluster sleep mode considerations
    • In version 0.23.x, the platform does not know how to wake a vCluster that has put itself to sleep. Ensure the vCluster is resumed before installing the agent to take over.
    • In version 0.24.x, the autoDelete configuration is still under external.platform and is set to move under sleepMode in a later release. external.platform to be moved under sleepMode in a future release.

    Ingress configuration​

    Sync to host​

    If you install ingress controllers in the vCluster, prevent them from sleeping. This keeps the controllers active and allows them to handle requests that would wake the vCluster. If you install the ingress controller in the host cluster, enable ingress syncing to the host.

    Enable ingress syncing
    sync:
      toHost:
        ingresses:
          enabled: true

    Reachability for ingress activity detection​

    To enable ingress activity detection in your vCluster, ensure the following conditions are met:

    • The ingress controller needs to be able to discover the vCluster pod to route traffic correctly.
    • Ensure that the DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local resolves properly. This allows the ingress controller to reach the vCluster's service address.
    • Ensure that the correct dnsPolicy is set in the ingress controller's Helm chart. This ensures DNS resolution works as expected and allows communication between the ingress controller and the vCluster.

    Configure sleep mode in a virtual cluster​

    The following examples demonstrate how to configure sleep mode in a virtual cluster.

    Ensure you install the necessary prerequisites.

    Prerequisites​

    • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

      info

      To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

    • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

    • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

    • vCluster: vCluster command-line tool to provision and manage virtual clusters. 
      brew install loft-sh/tap/vcluster

      The binaries in the tap are signed using the Sigstore framework for enhanced security.

      Confirm that you've installed the correct version of the vCluster CLI.

      vcluster --version
    • docker: Platform for building and running containerized applications.
    • kind: Tool for running local Kubernetes clusters in Docker.
    • curl: Command-line tool for transferring data over the network.

    Deployment​

    A deployment resource in Kubernetes manages a set of identical pods. Configuring sleep mode for a deployment scales down the pods while keeping the control plane active. This setup allows the virtual cluster to reduce resource usage while still being able to monitor activity and trigger wake-up actions when needed.

    Configure sleep mode with a deployment resource

    Configure sleep mode for a deployment resource​

    1. Create the kind cluster.

      create kind cluster
      kind create cluster --name sleep-mode-demo

    2. Deploy a virtual cluster.

      Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

      vCluster config for auto sleep
      pro: true
      sleepMode:
        enabled: true
        autoSleep:
          afterInactivity: 30s
          exclude:
            selector:
              labels:
                sleep: no-thanks

      And run the following command:

      Create vCluster with autoSleep config
      vcluster create my-vcluster -f vcluster.yaml

      Workloads with the label sleep: no-thanks don't enter sleep mode after 30 seconds.

    3. Create demo deployments in your virtual cluster.

      Use the following deployment YAML to create two deployments.

      Example deployments
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: sleepy-deployment
        labels:
          app: sleepy-dep
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: demo-dep-1
        template:
          metadata:
            labels:
              app: demo-dep-1
          spec:
            containers:
              - command:
                  - /agnhost
                  - serve-hostname
                  - --http=true
                  - --port=8080
                image: registry.k8s.io/e2e-test-images/agnhost:2.39
                name: sleepy-demo

      ---

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: no-sleep-deployment
        labels:
          sleep: no-thanks
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: demo-dep-2
        template:
          metadata:
            labels:
              app: demo-dep-2
          spec:
            containers:
              - command:
                  - /agnhost
                  - serve-hostname
                  - --http=true
                  - --port=8080
                image: registry.k8s.io/e2e-test-images/agnhost:2.39
                name: not-sleepy-demo

      The first deployment does not have any special configurations for sleep mode. You can replace it with another deployment if needed. The second deployment includes a special label on the Deployment, preventing it from scaling down after 30 seconds.

      You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

    4. Verify Deployments sleep status.

      deployment sleep check
      > sleep 30; kubectl get deployments
      NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
      default       no-sleep-deployment   2/2     2            2           1m
      default       sleepy-deployment     0/2     0            0           1m

      The sleepy-deployment reports 0/2 replicas after 30 seconds. Running kubectl counts as cluster activity, which is why it reports 0/2 instead of 0/0. The kubectl command triggers vCluster to update the replica count back to the original value of 2, but the replicas haven't become ready by the time kubectl get ... returns.

    Next steps​

    Experiment with the sleep mode feature by trying the following:

    • Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
    • Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

    Ingress controller​

    An ingress controller, such as NGINX, manages external HTTP/S traffic to services in a Kubernetes cluster. To configure sleep mode for the ingress controller, make sure it stays responsive to incoming traffic and can wake up the cluster if needed. This setup keeps the ingress controller active, even when the vCluster is in sleep mode and allows the controller to handle requests that trigger the virtual cluster to wake up.

    Configure sleep mode with an ingress resource

    Configure sleep mode for an ingress controller​

    1. Create the kind cluster.

      Create a kind cluster
      kind create cluster --name ingress-demo --config - <<EOF
      kind: Cluster
      apiVersion: kind.x-k8s.io/v1alpha4
      networking:
        apiServerAddress: "0.0.0.0"
      nodes:
        - role: control-plane
          extraPortMappings:
            - containerPort: 80
              hostPort: 80
              protocol: TCP
      EOF

    2. Install the NGINX IngressController.

      install ingress controller
      helm install ingress-nginx ingress-nginx/ingress-nginx \
        --namespace ingress-nginx \
        --create-namespace \
        --set controller.dnsPolicy=ClusterFirstWithHostNet \
        --set controller.hostNetwork=true \
        --set controller.service.type=ClusterIP

    3. Create the vCluster.

      Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

      vCluster config for auto sleep
      pro: true
      sync:
        toHost:
          ingresses:
            enabled: true
      sleepMode:
        enabled: true
        autoSleep:
          afterInactivity: 30s

      And run the following command:

      Create vCluster with autoSleep config
      vcluster create my-vcluster -f vcluster.yaml

    4. Enable local DNS resolution for the virtual cluster.

      Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

    5. Create resources for the Ingress such as a Deployment and Service.

      Use the following manifest to create:

      • Namespace named bar
      • Deployment for the pods backing the Service
      • Service for the Ingress
      • Ingress resource
      Example deployments
      apiVersion: v1
      kind: Namespace
      metadata:
        name: bar

      ---

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: bar-deployment
        namespace: bar
        labels:
          app: bar-dep
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: bar
        template:
          metadata:
            labels:
              app: bar
          spec:
            containers:
              - command:
                  - /agnhost
                  - serve-hostname
                  - --http=true
                  - --port=8080
                image: registry.k8s.io/e2e-test-images/agnhost:2.39
                name: bar-app

      ---

      kind: Service
      apiVersion: v1
      metadata:
        name: bar-service
        namespace: bar
      spec:
        selector:
          app: bar
        ports:
          # Default port used by the image
          - port: 8080

      ---

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: example-ingress
        namespace: bar
      spec:
        ingressClassName: nginx
        rules:
          - http:
              paths:
                - pathType: Prefix
                  path: /bar
                  backend:
                    service:
                      name: bar-service
                      port:
                        number: 8080
            host: backend.local

    6. Verify the ingress is working properly with curl.

      Test the Ingress endpoint within the 30-second activity window by running curl --silent backend.local/bar. The pod name from the Deployment that responds is displayed.

    7. Allow the virtual cluster to go to sleep.

      Wait 30 seconds for the cluster to sleep, then run the curl command again. For convenience, run watch -d curl --silent backend.local/bar to repeatedly test the endpoint. On the first attempt, you’ll see the message Client sent an HTTP request to an HTTPS server. because an HTTP request was sent to the HTTPS wake endpoint. Subsequent requests display new pod names.

    Istio gateway​

    note

    Enable Istio integration for sleep mode to work with Istio resources.

    When Istio is installed on the host cluster and the Istio integration is enabled, the vCluster syncs Gateway and VirtualService resources to the host cluster.

    With sleep mode enabled, any traffic routed to a Service in the vCluster is tracked as activity and keeps the vCluster awake. If the vCluster is asleep, incoming traffic wakes it up.

    Configure an ingress gateway for cluster external access
    info

    This example continues from the Istio integration example.

    1. Complete the Istio integration example.

    2. Install the Istio ingress gateway controller to allow routing from outside the cluster.

      Save the following manifest as ingress.yaml:

      apiVersion: install.istio.io/v1alpha1
      kind: IstioOperator
      metadata:
        name: ingress
      spec:
        profile: empty # Do not install CRDs or the control plane
        components:
          ingressGateways:
          - name: istio-ingressgateway
            namespace: istio-ingress
            enabled: true
            label:
              istio: ingressgateway
        values:
          gateways:
            istio-ingressgateway:
              injectionTemplate: gateway

    3. Create the istio-ingress namespace and install the ingress gateway:

      kubectl create namespace istio-ingress                                                                                                      
      istioctl install -f ingress.yaml

    4. Create a Gateway resource that uses a selector matching the newly installed ingress gateway in your vCluster.

      apiVersion: networking.istio.io/v1
      kind: Gateway
      metadata:
        name: istio-sm-gateway
        namespace: test
      spec:
        # The selector matches the ingress gateway pod labels.
        # If you installed Istio using Helm following the documentation, this is "istio=ingress"
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 80
            name: http
            protocol: HTTP
          hosts:
          - "smdemo.local"

    5. Update the vCluster with sleepMode enabled.

      Use the following vcluster.yaml to update your virtual cluster on your host cluster with --upgrade. Save this file as vcluster.yaml

      vCluster config for auto sleep
      pro: true
      sleepMode:
        enabled: true
        autoSleep:
          afterInactivity: 30s
      integrations:
        istio:
          enabled: true

      Run the following command:

      Create vCluster with autoSleep config
      vcluster create <your-vcluster-name> -f vcluster.yaml --upgrade

    6. Enable local DNS resolution for the virtual cluster.

      Add 127.0.0.1 smdemo.local to your /etc/hosts file to match the host configured in the Gateway configuration.

    7. Add the Gateway you created and the smdemo.local host to the VirtualService from the [Istio integration example](../vcluster- yaml/integrations/istio). The updated VirtualService should look like the following:

      Example resources
      apiVersion: networking.istio.io/v1
      kind: VirtualService
      metadata:
        name: nginx-service
        namespace: test
      spec:
        gateways:
          - istio-sm-gateway
        hosts:
          - smdemo.local
        http:
          - match:
              - uri:
                  prefix: /v2
            name: nginx-v2
            rewrite:
              uri: /
            route:
              - destination:
                  host: nginx-service.test.svc.cluster.local
                  subset: v2
          - name: nginx-v1
            route:
              - destination:
                  host: nginx-service.test.svc.cluster.local
                  subset: v1

    8. Use curl to verify that the Gateway is working correctly.

      Test the Gateway endpoint within the 30-second activity window by running either curl --silent smdemo.local/v1 or curl --silent smdemo.local/v2.

    9. Allow the virtual cluster to go to sleep.

      Wait 30 seconds for the cluster to sleep. Then run the curl command again. To repeatedly test the endpoint, run watch -d curl --silent smdemo.local/v2. While the cluster is asleep or waking up, a 503 error is displayed.

    Enable sleep mode with label selectors and schedules​

    Use label selectors and schedules to configure sleep mode based on inactivity or specific timing:

    Sleep mode label selectors and schedule
    Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 3h  # Uses Go duration with a max unit of hour
        exclude:
          selector:
            labels:
              dont: sleep
    Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
    sleepMode:
      enabled: true
      timeZone: America/Denver
      autoSleep:
        schedule: 30 17 * * 5
      autoWakeup:
        schedule: 0 7 * * 1

    Config reference​

    sleepMode required object ​

    SleepMode holds the native sleep mode configuration for Pro clusters

    enabled required boolean ​

    Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

    timeZone required string ​

    Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

    autoSleep required object ​

    AutoSleep holds autoSleep details

    afterInactivity required string ​

    AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

    schedule required string ​

    Schedule represents a cron schedule for when to sleep workloads

    exclude required object ​

    Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

    selector required object ​
    labels required object ​

    Labels defines what labels should be looked for

    autoWakeup required object ​

    AutoWakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

    schedule required string ​