Secret Management in Kubernetes

In this article, we’ll explore the different types of Kubernetes Secrets, methods of using them in Pods, advanced external secret management systems, and best practices to ensure your secrets are handled securely.

Types of Kubernetes Secrets

Kubernetes provides several built-in secret types, each designed to handle different forms of sensitive data. These include:

• Opaque Secrets: The most generic secret type. You can store any type of sensitive data encoded in base64.
• docker-registry (dockercfg) Secrets: Used to authenticate against a Docker registry, typically for pulling container images from private repositories.
• TLS Secrets: Used to store TLS certificates and private keys.

Opaque Secrets

Imperative configuration

kubectl create secret generic postgres-secret \
  --from-literal=username=admin \
  --from-literal=password='S3cr3tPassw0rd' \
  --namespace=secrets-demo

This command creates an Opaque Secret with a username and password. The values are base64-encoded by Kubernetes.

Declarative configuration

echo -n 'admin' | base64
# YWRtaW4=
echo -n 'S3cr3tP@ssw0rd' | base64
# UzNjcjN0UGFzc3cwcmQ=

# vi postgres-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
  namespace: secrets-demo
type: Opaque
data:
  username: YWRtaW4=
  password: UzNjcjN0UGFzc3cwcmQ=


kubectl create -f postgres-secret.yaml

The secret values are base64 encoded; they are obscured but not truly secret. Anyone who can read the Secret can easily decode the base64-encoded values.

kubectl get secret postgres-secret --namespace=secrets-demo --template=\{\{.data.username}} | base64 -d

kubectl get secret postgres-secret --namespace=secrets-demo --output jsonpath="{.data.password}" | base64 -d

Docker Config Secret

kubectl create secret docker-registry private-registry-secret \
  --docker-server=https://registry.local \
  --docker-username=admin \
  --docker-password=Passw0rd1234 \
  --namespace=secrets-demo

This type of secret is used to provide authentication for Private Docker Registries.

TLS Secret

kubectl create secret tls demo-local-tls-secret \
  --cert=./secrets-demo/demo-local-tls.crt \
  --key=./secrets-demo/demo-local-tls.key \
  --namespace=secrets-demo

This creates a TLS Secret from an existing certificate and private key.

Other secret type includes Service Account Token Secret, Basic authentication Secret, SSH authentication Secrets and Bootstrap token Secrets.

Using Secrets in Pods

Kubernetes allows you to expose secrets to your applications in various ways. The most common methods are:

• Using Secrets as Environment Variables
• Using Secrets as Volume Mounts

Using Secrets as Environment Variables

Secrets can be injected into a container’s environment variables, which is useful for configuration settings like API keys or database credentials.

Syntax:

env:
- name: ENV_VAR_NAME
  valueFrom:
    secretKeyRef:
      name: secret-name
      key: key-name

Example:

apiVersion: v1
kind: Pod
metadata:
  name: postgres
  namespace: secrets-demo
  labels:
    app: postgres
spec:
  containers:
  - name: postgres
    image: postgres
    env:
    - name: POSTGRES_USER
      valueFrom:
        secretKeyRef:
          name: postgres-secret
          key: username
    - name: POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: postgres-secret
          key: password

In this example, the environment variables POSTGRES_USERNAME and POSTGRES_PASSWORD are populated with the values from the postgres-secret secret.

Validate:

kubectl exec postgres --namespace=secrets-demo -- env PGPASSWORD=S3cr3tPassw0rd psql admin -U admin -h localhost -p 5432 -c "\l"

Using Secrets as Volume Mounts

Secrets can also be mounted as files into a container, which can be particularly useful for certificates or SSH keys.

Syntax:

containers:
  volumeMounts:
    - name: secret-volume-name
      mountPath: /mount/path
      readOnly: true
volumes:
  - name: secret-volume-name
    secret:
      secretName: secret-name

Example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: secrets-demo
spec:
  containers:
  - name: nginx
    image: nginx
    env:
      - name: NGINX_TLS_CRT
        value: /etc/nginx/ssl/tls.crt
      - name: NGINX_TLS_KEY
        value: /etc/nginx/ssl/tls.key
    volumeMounts:
     - name: nginx-certs
       mountPath: /etc/nginx/ssl
       readOnly: true
  volumes:
    - name: nginx-certs
      secret:
        secretName: demo-local-tls-secret

Here, the secret is mounted at /etc/secret-volume inside the container as files, where each secret key is represented as a file in that directory.

Validate:

kubectl exec nginx --namespace=secrets-demo -- cat /etc/nginx/ssl/tls.crt
kubectl exec nginx --namespace=secrets-demo -- cat /etc/nginx/ssl/tls.key
kubectl exec nginx --namespace=secrets-demo -- printenv
kubectl exec nginx --namespace=secrets-demo -- sh -c 'echo $NGINX_TLS_CRT'

Passing Secrets to Container Command Arguments

You can directly pass secrets as arguments to the container's command, which might be required by certain command-line tools or scripts.

In this example, the secret value POSTGRES_PASSWORD is passed directly to the container’s command.

Syntax:

env:
  - name: DEMO_PASSWORD
    valueFrom:
      secretKeyRef:
        name: secret-name
        key: password
command: ["--password", ":$(DEMO_PASSWORD)"]

Example:

apiVersion: v1
kind: Pod
metadata:
  name: mlflow
  namespace: secrets-demo
spec:
  imagePullSecrets:
    - name: private-registry-secret
  containers:
  - name: mlflow
    image: registry.local/mlflow:v1
    env:
      - name: POSTGRES_USER
        valueFrom:
          secretKeyRef:
            name: postgres-secret
            key: username
      - name: POSTGRES_PASSWORD
        valueFrom:
          secretKeyRef:
            name: postgres-secret
            key: password
      - name: POSTGRES_DATABASE
        value: admin
    command: ["mlflow", "server", "--backend-store-uri", "postgresql+psycopg2://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@postgres-service:5432/$(POSTGRES_DATABASE)"]

Rotating secrets in Kubernetes is essential for maintaining the security of your application by ensuring that sensitive information like credentials and keys remains protected. Kubernetes simplifies the process by allowing secret updates without causing downtime. The first step is to update the secret with new values, followed by redeploying the pods that rely on the secret. This can be done by either manually deleting the pods or updating the deployment, ensuring that the pods restart and pick up the updated secret values, enhancing the security of your system.

While Kubernetes secrets are base64-encoded, they are still stored in etcd (the underlying data store) in plaintext. For enhanced security and flexibility, integrate with external secret management systems like HashiCorp Vault (Vault Secrets Operator), AWS Secrets Manager (External Secrets Operator) or use Bitnami's Sealed Secrets Operator. (Followup article)

Secret management in Kubernetes is a critical aspect of maintaining application security. By understanding the different types of Kubernetes secrets, learning how to use them effectively, and implementing best practices like external secret managers, Sealed Secrets, and proper RBAC, you can securely manage and use secrets in your Kubernetes environments.

Following these guidelines will ensure that sensitive data remains secure, reducing the risk of exposure in dynamic and complex cloud-native environments.