Skip to main content
Version: main 🚧

Production and security

This guide covers the security practices that matter most for production Metal3 deployments. The four main risk areas are BMC credential handling, OS image integrity, platform role-based access control (RBAC), and the tenant isolation boundary.

BMC credential management​

Store BMC credentials​

The Metal3 Bare Metal Operator (BMO) connects to each server's BMC using credentials stored in a Kubernetes Secret. Every BareMetalHost references this Secret by name with spec.bmc.credentialsName. The Secret must be in the same namespace as the BareMetalHost on the control plane cluster.

The Secret requires exactly two keys:

apiVersion: v1
kind: Secret
metadata:
name: server-01-bmc-credentials
namespace: <metal3-namespace>
type: Opaque
stringData:
username: admin
password: <redacted>

Use a distinct Secret per server or per logical group of servers. A single shared Secret across all BareMetalHosts means one rotation event affects all of them at once, and makes it harder to scope access.

Rotate BMC credentials​

Rotating a BMC credential doesn't modify the OS or workloads on an already-provisioned server. BMO still uses the credential to observe and manage hardware state, so a mismatch can temporarily set the BareMetalHost's operational status to error and prevent power or lifecycle operations.

To rotate, update the BMC password on the server's BMC management interface, then update the Kubernetes Secret with the new password. Keep the interval between those changes as short as possible. BMO picks up the new credentials during reconciliation; you don't need to restart its Deployment or deprovision the server.

Check the BareMetalHost after the Secret update:

kubectl get baremetalhost <name> -n <namespace>
kubectl describe baremetalhost <name> -n <namespace>

An error does not move the host into a separate provisioning state. BMO leaves it in its current provisioning state and retries reconciliation. If the error remains, verify the Secret contents and BMC reachability before attempting another lifecycle operation.

Disable certificate verification only as a last resort​

The spec.bmc.disableCertificateVerification: true field tells Ironic to skip TLS certificate validation when connecting to the BMC over HTTPS. Don't set this in production: it allows a man-in-the-middle to intercept BMC traffic, including credentials.

If your BMC uses a self-signed certificate or an internal CA, add that CA to Ironic's BMC trust directory instead of disabling verification. The Ironic image combines certificates mounted under /certs/ca/bmc into the CA bundle it uses for BMC connections. Mount the CA Secret through the NodeProvider's metal3.deploy.metal3.helmValues field:

spec:
metal3:
deploy:
metal3:
enabled: true
helmValues: |
ironic:
extraVolumeMounts:
- name: bmc-ca
mountPath: /certs/ca/bmc
readOnly: true
extraVolumes:
- name: bmc-ca
secret:
secretName: bmc-ca-bundle

Create the bmc-ca-bundle Secret in the Metal3 release namespace on the control plane cluster, with the PEM-encoded CA certificate under key ca.crt. This is the namespace configured by the NodeProvider's clusterRef. Don't set IRONIC_CACERT_FILE for this purpose; that variable configures trust for the Ironic API rather than the BMC endpoints.

Control BMC proxy access​

The Metal3 UI is restricted to platform admins outside tenant scope. Within that view, Redfish Explorer checks permission to create metal3bmcs/redfish in the cluster.loft.sh API group. Scope access to the intended control plane cluster and BareMetalHost namespace.

The proxy uses the BMC credentials referenced by the host. A user doesn't need direct read access to that Secret to send an authorized request. Treat this permission as hardware administration access. It permits both reads and changes, including actions that can disrupt workloads.

The Redfish proxy verifies BMC TLS certificates unless the BareMetalHost sets spec.bmc.disableCertificateVerification: true.

Keep metal3bmcs/redfish requests at Metadata level in custom audit policies. BMC requests and responses can carry passwords or session tokens. See Audit BMC requests and the audit policy reference.

Trust the OS image pipeline​

Checksums are required​

When Ironic provisions a server, it downloads the OS image from spec.image.url. The checksum in spec.image.checksum ensures Ironic rejects corrupted or tampered images before writing them to disk, and aborts provisioning if the checksum doesn't match.

The checksum type defaults to MD5. Set spec.image.checksumType to sha256 or sha512 for stronger verification:

spec:
image:
url: https://images.internal/ubuntu-22.04.raw.gz
checksum: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
checksumType: sha256

TLS verification for OS images​

The platform sets the Metal3 Helm value ipaImages.insecure: true by default. Ironic and the IPA then skip TLS certificate verification for every OS image they fetch, not only images uploaded to the platform. This lets uploaded images work when the platform serves a self-signed certificate.

With verification off, the checksum is the only integrity check on the image, so its strength matters. Images uploaded to the platform record a sha256 checksum. For images referenced by URL, use a sha256 or sha512 checksum rather than MD5.

Turn verification back on when the platform presents a certificate from a public CA. The object storage behind it must do the same or serve plain HTTP. The Helm value is listed on the Metal3 & Ironic tab of Infrastructure deployment.

Host OS images on infrastructure you control​

Ironic fetches images at provisioning time from the control plane cluster. If the image URL is external, a network outage or a change at the remote host blocks all provisioning. Host OS images on infrastructure you control, such as an internal artifact registry or object storage on a private endpoint.

Node types reference OSImage resources by name. Control who can change an OSImage or a node type's image selection. Both affect which image the platform installs on the next provision.

Image signing​

Ironic doesn't natively verify image signatures. Checksum verification confirms integrity against a known-good value but doesn't prove provenance. Where image provenance matters, mitigate this by generating checksums in your image build pipeline and storing them alongside the image in your artifact registry, restricting write access to the image storage so that only your build system can publish new images, and restricting which OS image names and URLs can be referenced by the OSImage resource.

RBAC and resource boundaries​

Which roles control what​

NodeProvider and NodeType resources are cluster-scoped and non-namespaced. Platform admins manage their configuration. Tenants can read a reduced view of resources granted to them. That read access doesn't grant permission to change provider configuration. See Reduced read view.

Project roles determine what members of a Project can do with NodeClaim (instance) resources. See Project Membership for the default Project Admin, Project User, and Project Viewer roles. No Project role grants permission to change the infrastructure configuration behind an instance.

RBAC answers what an actor may do. Tenant and Project resource boundaries answer which infrastructure resources the actor may see or consume. Effective access is the intersection of role permissions, the Tenant boundary, and the Project boundary.

Restrict which node types a project can claim​

A Platform administrator defines the maximum set of node types available to a Tenant. A Tenant administrator can narrow that set for a Project with spec.allowedNodeTypes. A Project can't restore access to a node type outside its Tenant boundary. See Manage allowed node types for Project-level examples.

NodeProfile resources are cluster-scoped and shared across projects, so restrict them the same way with spec.allowedNodeProfiles. Because profiles set node labels, annotations, and taints, they influence scheduling inside the cluster. Treat the allow list as part of the tenant boundary rather than a convenience. See Manage allowed node profiles.

Prevent cross-project server access​

Each instance claim maps to a NodeClaim resource in the project's namespace (loft-p-<project>). The platform labels the claimed BareMetalHost with metal3.vcluster.com/node-claim: <nodeclaim-name> and vcluster.com/project: <project>. On the other side of the mapping, the NodeClaim annotation metal3.vcluster.com/claimed-node records the BareMetalHost name. The platform controller sets this metadata. Project users can't edit it.

BareMetalHost resources live on the control plane cluster, not in any tenant namespace. Tenants can't directly list or modify them. The platform mediates all access through NodeClaim objects, which are namespaced to the project.

Tenant isolation boundaries​

What the Metal3 driver provides​

Each instance claims one physical server. The platform allocates that server exclusively to one instance and cluster at a time. No other instance shares its CPU, memory, or storage, and no hypervisor layer sits between the workload and the hardware. This is physical compute isolation, unlike the tenancy model shared nodes provide on a control plane cluster's node pool.

This physical allocation is separate from management-plane resource boundaries. Tenant and Project policy controls who may consume the instance or its node type. The Metal3 driver enforces the lifecycle of the claimed server.

This model suits workloads with strict requirements for physical resource predictability, PCIe device exclusivity (GPUs, FPGAs, DPUs), or regulatory requirements that prohibit shared compute.

What the Metal3 driver doesn't provide​

Network isolation on the provisioning network. Before a server is claimed, and while it's mid-provisioning, it shares a provisioning network segment with every other server awaiting provisioning, regardless of which project eventually claims it. A compromised server can observe PXE traffic from other servers on this network during that window. Segment the provisioning network from production tenant traffic and treat it as an operations-only network. See Network configuration.

Isolation between workloads on the same node. Once a server joins a cluster, it's a standard Kubernetes worker node, and every pod scheduled onto it shares that node's kernel. This is true of any Kubernetes node, not a gap specific to Metal3 or vMetal, but it's worth calling out here because a bare metal server is often the sole node backing capacity for its tenant's own workloads: if those workloads don't trust each other (for example, running third-party or user-supplied code such as inference model containers), the boundary between them is the same one described in the shared nodes suitability guidance, and the same mitigation applies.

This concern doesn't arise for an independently provisioned instance, since no scheduler places other tenants' workloads on it. The boundary there is simpler and blunter. Whoever holds an SSH key on that node type has the whole server.

When to add vNode​

Add vNode as the container runtime when workloads scheduled onto a Metal3-provisioned node need privileged capabilities (Docker-in-Docker, hostPID, raw device access) but you don't fully trust what they do with those capabilities, for example because they run user-supplied inference code. vNode uses Linux user namespaces and seccomp to sandbox pods at the node level, so a container running as root inside vNode doesn't have real root on the host kernel. This hardens workloads on a single node. It doesn't change the physical isolation Metal3 provides by dedicating one server per instance.

When physical isolation is the right answer​

Physical dedication (one instance per server) is already Metal3's default. It's the right answer, with no further mitigation needed, when workloads require direct kernel access that vNode can't proxy (such as custom kernel modules or bare metal DPDK), when compliance requirements mandate physical isolation and disallow any software-layer sharing, or when performance isolation requirements are absolute, with no noisy-neighbor risk at the kernel scheduler level.

Harden the control plane cluster​

The control plane cluster that runs Metal3 and Ironic has a uniquely elevated trust level. The Ironic container holds BMC credentials and can power cycle, reimage, or wipe every registered server. Treat access to this cluster as equivalent to physical access to your server room.

Limit access to BareMetalHost resources​

Limit BareMetalHost writes to the driver controllers and authorized infrastructure operators. The platform agent needs read access to BareMetalHosts and their BMC credential Secrets to serve BMC requests. It doesn't need write access, because UI operations impersonate the requesting user. That operator needs the underlying resource permissions for each operation. For example, adding a server creates a BareMetalHost and, when BMC credentials are supplied, a credential Secret.

Scope direct operator access to the Metal3 namespace. For example, define this Role and bind it only to identities authorized to manage hosts:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: bmh-admin
namespace: metal3-system
rules:
- apiGroups: ["metal3.io"]
resources: ["baremetalhosts"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

This Role covers BareMetalHosts only. Grant access to credential Secrets separately when needed. Keep tenant identities out of both grants, and audit direct operator access.

Segment the control plane cluster's network​

The most effective way to restrict BMC network access is to dedicate specific control plane cluster nodes to Metal3 and Ironic, and block other workloads from scheduling there.

Taint the nodes that have BMC network access:

kubectl taint nodes <node-name> dedicated=metal3:NoSchedule
kubectl label nodes <node-name> dedicated=metal3

Configure the Metal3 deployment to schedule only on those nodes using the NodeProvider's metal3.deploy.metal3.helmValues:

deploy:
metal3:
enabled: true
helmValues: |
tolerations:
- key: dedicated
operator: Equal
value: metal3
effect: NoSchedule
nodeSelector:
dedicated: metal3

These settings constrain Metal3 scheduling. Taints don't enforce network isolation, and workloads with matching tolerations can still run on those nodes. Enforce BMC reachability with network controls too. Include the platform agent when enabling Redfish Explorer.

Enable audit logging on the control plane cluster​

This is separate from the platform's own audit logging feature, which records activity through the platform API. It doesn't see direct kubectl access to the control plane cluster's own API server, so enable native Kubernetes audit logging there as well, with a policy that captures write operations on both secrets and baremetalhosts:

rules:
- level: Metadata
resources:
- group: ""
resources: ["secrets"]
- level: Request
resources:
- group: "metal3.io"
resources: ["baremetalhosts"]
verbs: ["create", "update", "patch", "delete"]

Forward audit logs to your SIEM or log aggregation system. Alert on BareMetalHost writes outside the expected controller identities and approved operator activity.

Minimize human access​

Treat the control plane cluster as an operations cluster, not a development cluster. Human access should be limited to platform operators by identity rather than role, time-bounded where possible (break-glass access with an audit trail), and prohibited for tenant users entirely.

Don't attach the control plane cluster to vCluster Platform as a cluster that tenants can target for workloads. The cluster exists to run infrastructure components, not to host tenant environments.