Member post originally published on the CyberArk blog by John Walsh

Handling secrets in cloud-native environments is a challenge for many organizations. Virtually any application requires some sort of secret, such as a database password, a service token, or a certificate, or to establish secure connections. The growing popularity of Kubernetes in the last few years means that many security administrators are now searching for the capabilities to properly handle secrets in Kubernetes services. However, built-in secret capabilities of Kubernetes are limited: they lack certain functions that are essential for securely handling secrets.

Managing secrets insecurely poses many risks. Secrets leaked in source code, in application logs, or in another way, may fall into the wrong hands, with potentially disastrous results.

How Kubernetes handles secrets

Kubernetes provides a Secret object to securely store secret values. There are two types of secrets in Kubernetes:

Kubernetes secrets can be created in one of the following ways:

kubectl create secret my-secret –from-file=./credentials.txt

apiVersion: v1 kind: Secret metadata: name: my-secret data: username: bXktdXNlcm5hbWUK password: bXktcGFzc3dvcmQK type: Opaque

This object is then created based on the manifest using kubectl apply.

Limitations of Kubernetes secrets

There are several limitations to the method Kubernetes uses for storing secrets.

First, as the sensitive values in the example manifest above are base64 encoded – and thus not encrypted – it is easy to decode these values with the following command:echo bXktdXNlcm5hbWUK | base64 –decode my-username

Therefore, the secret’s manifest cannot be stored beside any other manifests (Deployments, Services, and so on) in Git. You don’t want to store a plain-text sensitive value in your repository!

Second, Kubernetes uses etcd as the storage for the secrets. Etcd is a distributed key/value storage known for its great performance. However, it lacks some features that are vital for sensitive data, such as an audit log, insights into key age, and automatic key rotation.

In addition, by default, Kubernetes stores secrets in etcd unencrypted, which means that anyone with access to the etcd cluster has access to the secrets.

Third, anyone with root access to the Kubernetes nodes can access all secrets by impersonating kubelet. Kubernetes currently shares all secrets with all nodes. Preventing privilege escalation is, therefore, essential for blocking access to secrets.

Fourth, Kubernetes can make secrets available to pods either through exposing them as environment variables or by mounting them as files containing plain-text secrets. Environment variables can easily expose sensitive values through, for example, debug logs. Plain-text files on disk are available to everyone with access to the pod.

Finally, Kubernetes’ RBAC functionality provides only get and set permissions for secrets. When you get a secret, you can only get its decrypted value. A more secure zero-trust setup would allow a developer to set a secret and then only retrieve the encrypted value for consumption. This is not available natively in Kubernetes.

Protecting secrets in container environments

A common approach to getting more secure secret management on Kubernetes is to introduce an external secret management solution, such as Hashicorp VaultAWS Secrets ManagerAzure Key Vault, or Google Secret Manager. While this approach resolves some issues inherent in Kubernetes, other – more general – secret management challenges persist. With a third-party solution, it is still virtually impossible to ensure that:

In addition, many third-party solutions don’t solve the Secret Zero problem.

Kubernetes secrets and CyberArk Conjur

The best way to deal with a secret is to… not deal with it at all. CyberArk Conjur can take care of it for you.

Conjur uses a sidecar container called Secretless Broker to proxy authenticated requests to external systems such as databases. The broker authenticates with CyberArk Conjur natively, using multi-factor authentication, to get access to the required secrets. This essentially resolves  the Secret Zero issue. Secret management is handled completely separately from the application, which eliminates the risk of leaking the secret. The application simply has no access to the secret!

The Secretless Broker never stores secrets on disk, and uses strong encryption in all communication with CyberArk Conjur, as well as in connections to the protected external systems.