Skip to main content
Version: main 🚧

Audit Logging

Enterprise
Available in these plansFreeDevProdScale
Audit Logging

Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.

Overview​

Audit logging in the platform provides a security-relevant, chronological set of records documenting the sequence of actions. It audits activities generated by users and applications that use the platform API.

The platform can log activities related to:

  • Management instance changes, such as creation of new clusters, spaces, and so on.
  • Changes within a clusterClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Cluster or space
  • Changes within a connected cluster
  • Sign-ins through password, SSO, token exchange, and the platform's OIDC provider, including failed attempts. See Authentication events.

Auditing in the platform is similar to auditing Kubernetes clusters in general.

Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.

Each request can be recorded with an associated stage. The defined stages are:

  • RequestReceived: The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
  • ResponseComplete: The response body has been completed and no more bytes are sent.
  • Panic: Events generated when a panic occurred.

The audit logging feature increases the memory consumption of the platform because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.

Enable auditing​

Two separate audit fields

values.yaml has two distinct audit fields with different purposes:

  • audit (top-level Helm value) configures the audit sidecar and log persistence (PVC) for the platform pod. This is a Helm-only setting. Changes require a Helm upgrade and a platform restart.
  • config.audit (under the config section) controls audit recording and policy, and enables the Audit Log UI. You can set it in Platform > Platform Config > Config in the UI or under config in values.yaml. config.audit.enabled defaults to true, so audit events are recorded without any additional configuration.

Changes to config.audit made through the UI take effect automatically within about 10-15 seconds, with no restart required. Changing config.audit through a Helm upgrade also restarts the platform pod, since the chart stamps the pod template with a config hash. The setting itself still takes effect through the same hot-reload mechanism either way.

Configure auditing through the platform config in the platform UI (Platform > Platform Config > Config). Content added in the UI goes automatically under the config section of values.yaml.

An example configuration could look like:

Example audit configuration in platform UI
audit:
enabled: true
level: 1

See the complete list of fields under the config.audit section of values.yaml.

Audit levels​

The platform provides audit levels, which are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most:

  • Level 1: Logs modifying requests such as creation, modification, or deletion of any objects. Creation requests also log the request payload.
  • Level 2: Like Level 1 but also logs the metadata of reading requests, such as listing pods inside a cluster or space. Reading requests won't log the response or request payload and instead only the metadata, such as request origin and target.
  • Level 3: Like Level 2 but instead of only logging the request metadata, also logs the complete request payload sent to the platform.
  • Level 4: Like Level 3 but instead of only logging metadata and request payload, also logs the response the platform has sent to the requester.

Captured payloads are normalized to JSON and size-limited before storage. Unconvertible or oversized payloads are omitted with an annotation. See Request and response payload storage.

Every level also applies a fixed set of exclusions and downgrades for internal, high-volume, or credential-carrying requests, on top of the level's own rules. See Default policy exclusions for what's affected and why. A custom audit policy is the only way to change this behavior, and it replaces these defaults entirely rather than layering on top of them.

Configure the audit level through the platform config, which can be modified either through the platform UI or Helm:

  1. Go to the Platform > Platform Config > Config view using the menu on the left.

  2. In the input field that appears, enter the following config:

    audit:
    enabled: true
    level: 1
  3. Click on the button. The change takes effect automatically within about 10-15 seconds, with no platform restart required.

Persist audit logs​

Persist platform audit logs in one of two ways: deploy the platform with a persistent volume claim (PVC), or connect the platform to a persistent database.

HA only supported with persistent database

If you are running the platform in high availability (HA mode), the only way to persist audit logs is to use a persistent database. The PVC approach doesn't work.

Deploy the platform with a PVC to save audit logs​

Create a new values.yaml with the following values:

values.yaml for PVC
audit:
persistence:
enabled: true
# size: 30Gi

This is the top-level audit field in values.yaml, not config.audit (see Enable auditing for the distinction).

Apply the values using Helm:

Apply Helm values
helm upgrade vcluster-platform vcluster-platform \
--namespace vcluster-platform \
--repo https://charts.loft.sh \
--reuse-values \
-f values.yaml

Use a persistent database as platform audit backend​

Instead of the internal sqlite storage (see View and export audit logs), the platform can write audit events to a persistent MySQL or PostgreSQL database. This is a config.audit setting, so you can configure it through the platform UI or Helm.

Through the UI, go to Platform > Platform Config > Config and specify the following platform config setting:

Platform config for persistent database
audit:
dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Through Helm, add the following to values.yaml:

values.yaml for persistent database
config:
audit:
enabled: true
dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

For PostgreSQL, use a postgres:// endpoint instead:

values.yaml for a PostgreSQL audit datastore
config:
audit:
enabled: true
dataStoreEndpoint: postgres://username:password@hostname:5432/database-name

Apply the changes. Through the UI, they take effect automatically within about 10-15 seconds, with no platform restart required. Through Helm, helm upgrade restarts the platform pod as a side effect of the upgrade, though the setting itself takes effect through the same hot-reload mechanism either way.

IAM authentication instead of a static password​

Set config.audit.dataStoreIdentityProvider: aws to authenticate with a short-lived IAM token instead of a static password. This works against a datastore with IAM database authentication enabled, such as RDS or Aurora. aws is the only supported value today, and it only applies to MySQL and PostgreSQL, not sqlite.

values.yaml for IAM-authenticated audit datastore
config:
audit:
enabled: true
dataStoreEndpoint: mysql://iam-user@tcp(hostname:3306)/database-name
dataStoreIdentityProvider: aws

Omit the password from dataStoreEndpoint when using IAM authentication. The platform mints a new token periodically to stay ahead of expiry.

Verify the datastore with a CA file​

Set config.audit.dataStoreCAFile to the path of a PEM-encoded CA bundle to verify the datastore's TLS certificate. The platform reads this file directly, so mount the Secret or ConfigMap containing it into the platform pod through the chart's top-level volumes and volumeMounts. This is the same approach as config.database's caFile:

values.yaml for a CA-verified audit datastore
volumes:
- name: audit-datastore-ca
secret:
secretName: audit-datastore-ca
volumeMounts:
- name: audit-datastore-ca
mountPath: /etc/vcluster/audit-datastore-ca
readOnly: true
config:
audit:
enabled: true
dataStoreEndpoint: postgres://username:password@hostname:5432/database-name
dataStoreCAFile: /etc/vcluster/audit-datastore-ca/ca.pem

The platform reads dataStoreCAFile once at startup. Rotating the mounted CA requires a platform pod restart to take effect.

TLS precedence

An explicit sslmode= (PostgreSQL) or tls= (MySQL) query parameter already present on dataStoreEndpoint always wins over dataStoreCAFile, silently. Remove the query parameter if you want dataStoreCAFile to take effect.

Without an explicit override:

  • PostgreSQL defaults to sslmode=require (encrypted, but the server certificate isn't verified). Setting dataStoreCAFile upgrades this to sslmode=verify-full.
  • MySQL has no TLS by default, unless dataStoreIdentityProvider is set, in which case it defaults to tls=skip-verify. Setting dataStoreCAFile upgrades either case to full certificate verification.

dataStoreCAFile applies regardless of dataStoreIdentityProvider, and has no effect on the internal sqlite backend.

View and export audit logs​

By default, the platform logs audit events to the following locations:

  • To a log file in JSON format located at /var/lib/loft/audit.log inside the platform container. Each line inside the log represents a single audit event.
  • To an internal sqlite storage located at /var/lib/loft/audit.db inside the platform container. This sqlite database is used to display audit log events in the platform UI. By default audit events in the sqlite are not persisted, so restarting the platform clears the database. See Use a persistent database as platform audit backend to configure a persistent MySQL backend instead.

Enable audit sidecar​

To export audit events to third-party systems, enable the audit log sidecar. It prints all audit events to stdout in a separate container, so you can watch and export them from there. You can only enable the sidecar through Helm values.

Create a values.yaml with the following contents:

values.yaml for audit sidecar
audit:
enableSideCar: true

This is the top-level audit field in values.yaml, not config.audit, so it cannot be set in Platform > Platform Config > Config (see Enable auditing for the distinction).

Update the Helm release:

Update Helm release
helm upgrade vcluster-platform vcluster-platform \
--namespace vcluster-platform \
--repo https://charts.loft.sh \
--reuse-values \
--values values.yaml

Wait until the platform has restarted, then view the audit logs:

View audit logs
kubectl logs -n vcluster-platform -l app=loft -c audit -f

Audit logging for Regional Cluster Endpoints​

Regional Cluster Endpoints allow direct communication between users and clusters, bypassing the central platform instance. When this feature is enabled, the platform audit configuration is synced to each agent. Each platform agent propagates audit events it receives back to the central platform instance, which then logs it as a regular audit event. These "propagated" events can be identified by the annotations.audit.loft.sh/sent-by-agent identifier in an audit event.

To disable this behavior, set the audit config option disableAgentSyncBack.