Skip to main content
Version: v0.27 Stable
Limited vCluster Tenancy Configuration Support

This feature is only available when using the following worker node types:

  • Host Nodes
    • Enterprise-Only Feature

    This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

    Sleep mode

    Not all workloads need to run continuously. Scaling them down saves time and reduces costs. With sleep mode, you can scale workloads based on a set schedule or user activity and ingress.

    warning

    Sleep mode is intended for pre-production use cases only, and has limitations when used on a standalone vCluster not connected to the platform.

    Enable sleep mode​

    To enable sleep mode, add the following configuration to your vcluster.yaml:

    Sleep mode configuration
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 1h

    Sleep mode operations​

    Sleep mode involves two main actions: sleeping and waking. These actions help save resources when the cluster is not in use.

    Sleeping reduces resource consumption and costs during inactive periods.

    • Delete bare pods: Pods not managed by controllers are removed permanently.
    • Scale down resources: The following resources are scaled down to reduce resource usage:
      • Deployments
      • ReplicaSets
      • ReplicationControllers
      • DaemonSets

    Waking restore scaled-down resources.

    • Restore scaled-down resources: Deployments, ReplicaSets, ReplicationControllers, and DaemonSets are returned to their original state.
    • Bare pods are not restored: Bare pods deleted during the Sleeping step cannot be recovered.

    Resource exemption in sleep mode​

    In sleep mode, you can prevent specific resources from entering sleep mode by configuring them with the following options:

    • Add the annotation sleepmode.loft.sh/exclude: true to the resource.
    • Configure sleepMode with specific labels to define which resources should remain active. This allows targeting specific resources based on their labels.
    • Add labels to workloads that you want to remain active. By using labels, it is possible to control which resources are exempt from sleep mode at a more granular level.

    Detect activity in sleep mode​

    Sleep mode tracks certain actions to detect activity and wake the cluster when necessary. To wake a sleeping cluster or update the last active time, sleep mode captures the following actions:

    • Accessing cluster resources through API calls (for example kubectl get <resource>).
    • Attempting to contact ingress endpoints (NGINX only).
    note

    Ingress activity detection works only with NGINX ingress controllers, using the mirror-target annotation. This overwrites any previously set mirror-target annotation.

    Ignore other types of activity in sleep mode​

    The resource exemption feature keeps specifically configured resources active when the vCluster enters sleep mode. The Configure request handling for sleep mode table describes how to ignore requests that keep the cluster awake or wake it up if it is asleep.

    Configure request handling for sleep mode table
    AnnotationBehaviorFormat
    sleepmode.loft.sh/ignore-allIgnore all activity.true or false
    sleepmode.loft.sh/ignore-ingressesIgnore requests to associated ingresses.true or false
    sleepmode.loft.sh/ignore-groupsIgnore specified groups the requesting user belongs to.group2,group3
    sleepmode.loft.sh/ignore-resourcesIgnore requests to specific resource types.pods,resource2,resource3
    sleepmode.loft.sh/ignore-verbsIgnore requests with specific verbs.create,verb1,verb2
    sleepmode.loft.sh/ignore-resource-verbsIgnore requests with specific verbs to specific resources.myresource.mygroup=create update delete, myresource2.mygroup=create update
    sleepmode.loft.sh/ignore-resource-namesIgnore requests to specific resources with specific names.myresource.mygroup=name1 name2,myotherresource.mygroup=name1
    sleepmode.loft.sh/ignore-user-agentsIgnore specific user agents with trailing wildcard support.kubectl*,argo,useragent3
    sleepmode.loft.sh/disable-ingress-wakeupIgnore ingress access.true
    note

    Add the annotations to the vCluster workload, such as the StatefulSet or Deployment running the virtual cluster. You can dynamically ignore any request by adding the X-Sleep-Mode-Ignore header.

    vCluster sleep mode compatibility with the platform​

    You can configure sleep mode in a virtual cluster in two ways:

    • Without an agent: This scales down workloads (like pods) while keeping the control plane active. The cluster remains in a minimal state and can wake up when activity is detected.

    • With an agent: When connected to the platform using an agent, the agent fully manages sleep mode by shutting down the control plane. This reduces resource usage more and enables features like auto-deletion, which aren’t available without the agent.

    When a vCluster connects to the platform using an agent, the agent manages sleep mode actions by shutting down the control plane completely. This reduces resource usage and enables features like auto-deletion, which is not available when the vCluster is not connected to the platform using an agent.

    Connect to the platform​

    When you connect a vCluster configured for sleep mode to the platform, it stops managing its own sleep mode actions and allows the agent to take control. This requires compatible versions of vCluster and the platform. The vCluster and platform version compatibility with agent installation table provides the steps needed for installing or uninstalling an agent.


    vCluster and platform version compatibility with agent installation table
    vCluster versionPlatform versionCompatibilityInstalling an agentUninstalling an agent
    0.22.x4.2.x✅Update the vCluster config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
    0.23.x4.2.x✅Update the vCluster config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
    0.24.0<4.3.0❌These versions are not compatible, as the vCluster version is ahead of the platform, which causes the vCluster creation to be rejected.Not applicable.
    0.24.0≥4.3.0✅No action is required. The platform reads the unified vCluster config and takes over as if it had been configured for the platform all along.Remove the annotation vcluster.loft.sh/agent-installed from the vCluster config secret in the host cluster to notify the vCluster that it needs to resume sleep mode. The secret’s name is vc-config-[vcluster-name].
    ≥0.24.0 (future release)≥4.3.0✅No action is required. The platform reads the unified config and takes over as if it had been configured for the platform all along.No action is required, and the vCluster resumes sleep mode for workloads only, not the control plane.
    vCluster sleep mode considerations
    • In version 0.23.x, the platform does not know how to wake a vCluster that has put itself to sleep. Ensure the vCluster is resumed before installing the agent to take over.
    • In version 0.24.x, the autoDelete configuration is still under external.platform and is set to move under sleepMode in a later release. external.platform to be moved under sleepMode in a future release.

    Ingress configuration​

    Sync to host​

    If you install ingress controllers in the vCluster, prevent them from sleeping. This keeps the controllers active and allows them to handle requests that would wake the vCluster. If you install the ingress controller in the host cluster, enable ingress syncing to the host.

    Enable ingress syncing
    sync:
      toHost:
        ingresses:
          enabled: true

    Reachability for ingress activity detection​

    To enable ingress activity detection in your vCluster, ensure the following conditions are met:

    • The ingress controller needs to be able to discover the vCluster pod to route traffic correctly.
    • Ensure that the DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local resolves properly. This allows the ingress controller to reach the vCluster's service address.
    • Ensure that the correct dnsPolicy is set in the ingress controller's Helm chart. This ensures DNS resolution works as expected and allows communication between the ingress controller and the vCluster.

    Configure sleep mode in a virtual cluster​

    The following examples demonstrate how to configure sleep mode in a virtual cluster.

    Ensure you install the necessary prerequisites.

    Prerequisites​

    • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

      info

      To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

    • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

    • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

    • vCluster: vCluster command-line tool to provision and manage virtual clusters. 
      brew install loft-sh/tap/vcluster

      The binaries in the tap are signed using the Sigstore framework for enhanced security.

      Confirm that you've installed the correct version of the vCluster CLI.

      vcluster --version
    • docker: Platform for building and running containerized applications.
    • kind: Tool for running local Kubernetes clusters in Docker.
    • curl: Command-line tool for transferring data over the network.

    Deployment​

    A deployment resource in Kubernetes manages a set of identical pods. Configuring sleep mode for a deployment scales down the pods while keeping the control plane active. This setup allows the virtual cluster to reduce resource usage while still being able to monitor activity and trigger wake-up actions when needed.

    Configure sleep mode with a deployment resource

    Ingress controller​

    An ingress controller, such as NGINX, manages external HTTP/S traffic to services in a Kubernetes cluster. To configure sleep mode for the ingress controller, make sure it stays responsive to incoming traffic and can wake up the cluster if needed. This setup keeps the ingress controller active, even when the vCluster is in sleep mode and allows the controller to handle requests that trigger the virtual cluster to wake up.

    Configure sleep mode with an ingress resource

    Istio gateway​

    note

    Enable Istio integration for sleep mode to work with Istio resources.

    When Istio is installed on the host cluster and the Istio integration is enabled, the vCluster syncs Gateway and VirtualService resources to the host cluster.

    With sleep mode enabled, any traffic routed to a Service in the vCluster is tracked as activity and keeps the vCluster awake. If the vCluster is asleep, incoming traffic wakes it up.

    Configure an ingress gateway for cluster external access

    Enable sleep mode with label selectors and schedules​

    Use label selectors and schedules to configure sleep mode based on inactivity or specific timing:

    Sleep mode label selectors and schedule
    Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 3h  # Uses Go duration with a max unit of hour
        exclude:
          selector:
            labels:
              dont: sleep
    Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
    sleepMode:
      enabled: true
      timeZone: America/Denver
      autoSleep:
        schedule: 30 17 * * 5
      autoWakeup:
        schedule: 0 7 * * 1

    Config reference​

    sleepMode required object ​

    SleepMode holds the native sleep mode configuration for Pro clusters

    enabled required boolean ​

    Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

    timeZone required string ​

    Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

    autoSleep required object ​

    AutoSleep holds autoSleep details

    afterInactivity required string ​

    AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

    schedule required string ​

    Schedule represents a cron schedule for when to sleep workloads

    exclude required object ​

    Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

    selector required object ​
    labels required object ​

    Labels defines what labels should be looked for

    autoWakeup required object ​

    AutoWakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

    schedule required string ​