Vault on Kubernetes security considerations

Vault is a security product responsible for protecting sensitive data and serves as a single source of secrets. The Production Hardening documentation provides recommendations based on the security model assuming that Vault is deployed on VMs instead of containers. When you deploy Vault on Kubernetes, those best practices are not quite transparent. This tutorial highlights where extra precaution is needed when you deploy Vault on Kubernetes in production.

The following topics are addressed in this tutorial:

• Single vs. Multi Tenancy Runtimes
• End-to-End TLS
• Turn Off Core Dumps
• Protect Vault memory from swap
• Container Supervisor
• Don't Run as Root

Single vs. Multi Tenancy Runtimes

The Vault Production Hardening documentation states that Vault should be the only main process running on a machine to reduce the surface area introduced by other tenants.

Limitations introduced by running Vault on Kubernetes

Consul Helm chart only allows one Consul server cluster per Kubernetes cluster. This makes Consul available to other tenants on the Kubernetes cluster which undermines the security model for Vault when using Consul as its storage backend.

Another security concern is that the Vault container is owned by root, but the Vault executable inside the container is still run as the vault user. This means that the supervisor process is owned by root. Any process that escapes the pod will have root privileges on the Kubernetes worker node. (Also, see the Container Supervisor section.)

End-to-End TLS

Vault should always be used with Transport Layer Security (TLS) in production. If intermediate load balancers or reverse proxies are used to front Vault, they should not terminate TLS to ensure that the traffic is always encrypted in transit.

Limitations introduced by running Vault on Kubernetes

Kubernetes uses TLS throughout the system to connect disparate components. However, Kubernetes does not verify TLS connections by default for certain connections, and portions of the codebase include the use of InsecureSkipVerify which precludes verification of the presented certificate.

Issue Example:

if dialer != nil {
    // We have a dialer; use it to open the connection, then
    // create a tls client using the connection.
    netConn, err := dialer(ctx, "tcp", dialAddr)
    if err != nil {
        return nil, err
    }
    if tlsConfig == nil {
        // tls.Client requires non-nil config
        klog.Warningf("using custom dialer with no TLSClientConfig. Defaulting to InsecureSkipVerify")
        // tls.Handshake() requires ServerName or InsecureSkipVerify
        tlsConfig = &tls.Config{
            InsecureSkipVerify: true,
        }
    // ...
}

Mitigations

• Use TLS certificates signed by a trusted Certificate Authority (CA)
• Use intermediate CA's to protect the root CA
• Verify the authenticity of the server when making client request
• Verify the authenticity of the client when responding to requests
• Require TLS 1.2+

Turn Off Core Dumps

A user or administrator that can force a core dump and has access to the resulting file can potentially access Vault encryption keys. Therefore, ensure that the process core dumps are disabled inside the container.

Limitations introduced by running Vault on Kubernetes

• Root user on the worker node can access procfs (process filesystem) for the containerized process
• In a multi-tenant cluster, containers running with the --privileged flag will be able to access procfs on the host machine
• If this is the same machine that the Vault container is running, a process from the privileged container could exfiltrate the key from a core dump of the Vault process in the container

Mitigations

Ensure RLIMIT_CORE is set to 0 or use the ulimit command with the core flag (ulimit -c 0) inside the container to ensure your container processes can't core dump.

Protect Vault memory from swap

Vault uses memory locking (mlock) to prevent sensitive data in process memory from being written to swap. The default Vault container image runs as the non-root vault user and does not include the cap_ipc_lock file capability on the vault binary. As a result, mlock() cannot succeed when Vault runs from the default image, regardless of the container runtime.

To allow Vault to start without mlock(), set disable_mlock to true in the Vault server configuration or set the VAULT_DISABLE_MLOCK=true as an environment variable.

The Vault Helm chart applies disable_mlock = true automatically when you do not configure the value explicitly.

If you must disable mlock, make sure to protect Vault process memory at the node level and only run Vault on nodes where you have disabled or encrypted swap storage. Kubernetes does not enable swap on Linux nodes by default, but you should verify the swap configuration on your platform before deployment.

Mitigations

• Schedule Vault on a node pool with swap disabled. Use node labels and nodeAffinity to require approved nodes. In mixed clusters, taint swap-enabled nodes and do not add matching tolerations to Vault pods.
• For nodes that must have swap enabled, verify that swap storage is encrypted.
• Do not add IPC_LOCK to the Vault pod security context for the default image. The capability has no effect, and admission controllers on some platforms reject pods that request it.
• We do not recommend creating custom images, but if your security policy requires mlock, you can build a custom Vault image with cap_ipc_lock set on the vault binary and IPC_LOCK capability added to the Vault container with a security policy that permits the IPC_LOCK capability:

securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    capabilities:
      add: ["IPC_LOCK"]

You cannot use the resulting container image with integrated storage or platforms that reject IPC_LOCK such as GKE Autopilot and AWS Fargate.

Container Supervisor

The supervisor process that starts your container should not run as root.

Limitations introduced by running Vault on Kubernetes

If your container starts as root, the processes that escape that container have root on the node they were started on.

Mitigations

Apply security context for your container and pod to prevent starting your container as root. (A security context is a property defined in the deployment yaml.)

Generic Kubernetes Pod example:

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
  # specification of the pod’s containers
  # ...
  securityContext:
    runAsNonRoot: true

Don't Run as Root

Vault is designed to run as an unprivileged user, and there is no reason to run Vault with root or administrator privileges which can expose the Vault process memory and allow access to Vault encryption keys.

Limitations introduced by running Vault on Kubernetes

Do not circumvent the entry point for your container. This will also circumvent the user to use PID 1 in the container and use root instead. (Refer to Issue #205.)

Mitigations

Follow the hardening guidelines and ensure your Vault process is not started as root user inside your container.

Help and Reference

• Vault Security Model
• Production hardening documentation