Audit Logging
| Enterprise | ||||
|---|---|---|---|---|
| Available in these plans | Free | Dev | Prod | Scale |
| Audit Logging | ||||
Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.
Overview​
Audit logging in the platform provides a security-relevant, chronological set of records documenting the sequence of actions. It audits activities generated by users and applications that use the platform API.
The platform can log activities related to:
- Management instance changes, such as creation of new clusters, spaces, and so on.
- Changes within a clusterClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster. or space
- Changes within a connected cluster
- Sign-ins through password, SSO, token exchange, and the platform's OIDC provider, including failed attempts. See Authentication events.
Auditing in the platform is similar to auditing Kubernetes clusters in general.
Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.
Each request can be recorded with an associated stage. The defined stages are:
RequestReceived: The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.ResponseComplete: The response body has been completed and no more bytes are sent.Panic: Events generated when a panic occurred.
The audit logging feature increases the memory consumption of the platform because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.
Enable auditing​
values.yaml has two distinct audit fields with different purposes:
audit(top-level Helm value) configures the audit sidecar and log persistence (PVC) for the platform pod. This is a Helm-only setting. Changes require a Helm upgrade and a platform restart.config.audit(under theconfigsection) controls audit recording and policy, and enables the Audit Log UI. You can set it in Platform > Platform Config > Config in the UI or underconfiginvalues.yaml.config.audit.enableddefaults totrue, so audit events are recorded without any additional configuration.
Changes to config.audit made through the UI take effect automatically within about 10-15 seconds, with no restart required. Changing config.audit through a Helm upgrade also restarts the platform pod, since the chart stamps the pod template with a config hash. The setting itself still takes effect through the same hot-reload mechanism either way.
Configure auditing through the platform config in the platform UI (Platform > Platform Config > Config). Content added in the UI goes automatically under the config section of values.yaml.
An example configuration could look like:
audit:
enabled: true
level: 1
See the complete list of fields under the config.audit section of values.yaml.
Audit levels​
The platform provides audit levels, which are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most:
- Level 1: Logs modifying requests such as creation, modification, or deletion of any objects. Creation requests also log the request payload.
- Level 2: Like Level 1 but also logs the metadata of reading requests, such as listing pods inside a cluster or space. Reading requests won't log the response or request payload and instead only the metadata, such as request origin and target.
- Level 3: Like Level 2 but instead of only logging the request metadata, also logs the complete request payload sent to the platform.
- Level 4: Like Level 3 but instead of only logging metadata and request payload, also logs the response the platform has sent to the requester.
Captured payloads are normalized to JSON and size-limited before storage. Unconvertible or oversized payloads are omitted with an annotation. See Request and response payload storage.
Every level also applies a fixed set of exclusions and downgrades for internal, high-volume, or credential-carrying requests, on top of the level's own rules. See Default policy exclusions for what's affected and why. A custom audit policy is the only way to change this behavior, and it replaces these defaults entirely rather than layering on top of them.
Configure the audit level through the platform config, which can be modified either through the platform UI or Helm:
- UI
- Helm
Go to the Platform > Platform Config > Config view using the menu on the left.
In the input field that appears, enter the following config:
audit:enabled: truelevel: 1Click on the button. The change takes effect automatically within about 10-15 seconds, with no platform restart required.
Create a new file values.yaml:
config:
audit:
enabled: true
level: 1
Then apply these values with Helm (make sure to specify the correct platform version):
helm upgrade loft vcluster-platform --namespace vcluster-platform \
--repo https://charts.loft.sh \
--version $LOFT_VERSION \
--reuse-values \
--values values.yaml
Persist audit logs​
Persist platform audit logs in one of two ways: deploy the platform with a persistent volume claim (PVC), or connect the platform to a persistent database.
If you are running the platform in high availability (HA mode), the only way to persist audit logs is to use a persistent database. The PVC approach doesn't work.
Deploy the platform with a PVC to save audit logs​
Create a new values.yaml with the following values:
audit:
persistence:
enabled: true
# size: 30Gi
This is the top-level audit field in values.yaml, not config.audit (see Enable auditing for the distinction).
Apply the values using Helm:
helm upgrade vcluster-platform vcluster-platform \
--namespace vcluster-platform \
--repo https://charts.loft.sh \
--reuse-values \
-f values.yaml
Use a persistent database as platform audit backend​
Instead of the internal sqlite storage (see View and export audit logs), the platform can write audit events to a persistent MySQL or PostgreSQL database. This is a config.audit setting, so you can configure it through the platform UI or Helm.
Through the UI, go to Platform > Platform Config > Config and specify the following platform config setting:
audit:
dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name
Through Helm, add the following to values.yaml:
config:
audit:
enabled: true
dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name
For PostgreSQL, use a postgres:// endpoint instead:
config:
audit:
enabled: true
dataStoreEndpoint: postgres://username:password@hostname:5432/database-name
Apply the changes. Through the UI, they take effect automatically within about 10-15 seconds, with no platform restart required. Through Helm, helm upgrade restarts the platform pod as a side effect of the upgrade, though the setting itself takes effect through the same hot-reload mechanism either way.
IAM authentication instead of a static password​
Set config.audit.dataStoreIdentityProvider: aws to authenticate with a short-lived IAM token instead of a static password. This works against a datastore with IAM database authentication enabled, such as RDS or Aurora. aws is the only supported value today, and it only applies to MySQL and PostgreSQL, not sqlite.
config:
audit:
enabled: true
dataStoreEndpoint: mysql://iam-user@tcp(hostname:3306)/database-name
dataStoreIdentityProvider: aws
Omit the password from dataStoreEndpoint when using IAM authentication. The platform mints a new token periodically to stay ahead of expiry.
Verify the datastore with a CA file​
Set config.audit.dataStoreCAFile to the path of a PEM-encoded CA bundle to verify the datastore's TLS certificate. The platform reads this file directly, so mount the Secret or ConfigMap containing it into the platform pod through the chart's top-level volumes and volumeMounts. This is the same approach as config.database's caFile:
volumes:
- name: audit-datastore-ca
secret:
secretName: audit-datastore-ca
volumeMounts:
- name: audit-datastore-ca
mountPath: /etc/vcluster/audit-datastore-ca
readOnly: true
config:
audit:
enabled: true
dataStoreEndpoint: postgres://username:password@hostname:5432/database-name
dataStoreCAFile: /etc/vcluster/audit-datastore-ca/ca.pem
The platform reads dataStoreCAFile once at startup. Rotating the mounted CA requires a platform pod restart to take effect.
An explicit sslmode= (PostgreSQL) or tls= (MySQL) query parameter already present on dataStoreEndpoint always wins over dataStoreCAFile, silently. Remove the query parameter if you want dataStoreCAFile to take effect.
Without an explicit override:
- PostgreSQL defaults to
sslmode=require(encrypted, but the server certificate isn't verified). SettingdataStoreCAFileupgrades this tosslmode=verify-full. - MySQL has no TLS by default, unless
dataStoreIdentityProvideris set, in which case it defaults totls=skip-verify. SettingdataStoreCAFileupgrades either case to full certificate verification.
dataStoreCAFile applies regardless of dataStoreIdentityProvider, and has no effect on the internal sqlite backend.
View and export audit logs​
By default, the platform logs audit events to the following locations:
- To a log file in JSON format located at
/var/lib/loft/audit.loginside the platform container. Each line inside the log represents a single audit event. - To an internal sqlite storage located at
/var/lib/loft/audit.dbinside the platform container. This sqlite database is used to display audit log events in the platform UI. By default audit events in the sqlite are not persisted, so restarting the platform clears the database. See Use a persistent database as platform audit backend to configure a persistent MySQL backend instead.
Enable audit sidecar​
To export audit events to third-party systems, enable the audit log sidecar. It prints all audit events to stdout in a separate container, so you can watch and export them from there. You can only enable the sidecar through Helm values.
Create a values.yaml with the following contents:
audit:
enableSideCar: true
This is the top-level audit field in values.yaml, not config.audit, so it cannot be set in Platform > Platform Config > Config (see Enable auditing for the distinction).
Update the Helm release:
helm upgrade vcluster-platform vcluster-platform \
--namespace vcluster-platform \
--repo https://charts.loft.sh \
--reuse-values \
--values values.yaml
Wait until the platform has restarted, then view the audit logs:
kubectl logs -n vcluster-platform -l app=loft -c audit -f
Audit logging for Regional Cluster Endpoints​
Regional Cluster Endpoints allow direct communication between users and clusters, bypassing the central platform instance. When this feature is enabled, the platform audit configuration is synced to each agent. Each platform agent propagates audit events it receives back to the central platform instance, which then logs it as a regular audit event. These "propagated" events can be identified by the annotations.audit.loft.sh/sent-by-agent identifier in an audit event.
To disable this behavior, set the audit config option disableAgentSyncBack.