External Secrets in Kubernetes

While Kubernetes offers its native secret management system, it’s not the best option for everyone, especially in scenarios where you need tighter security, centralized management, and audit controls. This is where external secret management comes into play.

Kubernetes Secrets provide a way to manage sensitive information, but they have limitations. The secrets are base64 encoded, which is not an encryption mechanism and can be easily decoded. Without proper encryption at rest and strict access controls, secrets are vulnerable to exposure.

External secret management solves this by storing sensitive data outside the Kubernetes cluster in external secret managers, such as AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or HashiCorp Vault. These tools are designed specifically to manage and protect secrets, offering features like encryption, access control policies, and auditing.

Bitnami Sealed Secrets

Sealed Secrets are a way to encrypt Kubernetes Secrets using a public key so that only the controller running in your cluster can decrypt them. This enables you to store secrets in version control safely.

Install Sealed Secrets Controller

Add the Bitnami Sealed Secrets Helm repository and install the Sealed Secrets controller in the kube-system namespace. The controller is responsible for decrypting the sealed secrets stored in the cluster.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets

Install client tool

Download and install the kubeseal CLI tool on your local machine for encrypting Kubernetes secrets into sealed secrets that can be safely stored in version control.

brew install kubeseal

Retrieve Public Certificate for Encryption

Fetches the public certificate used by the Sealed Secrets controller to encrypt secrets. The certificate is saved to a file for use in the encryption process.

kubeseal \
    --controller-name=sealed-secrets-controller \
    --controller-namespace=kube-system \
    --fetch-cert > sealed-secrets-cert.pem

Create a Sealed Secret file

Create a generic Kubernetes secret and pipe it to kubeseal, which encrypts it using the public certificate. The resulting sealed secret is saved to a YAML file that can be committed to version control safely and ideal for GitOps workflow.

kubectl create secret generic ss-postgres-secret --namespace=secrets-demo \
    --from-literal=username=admin \
    --from-literal=password='S3cr3tPassw0rd' \
    --dry-run=client -o yaml | \
    kubeseal --cert sealed-secrets-cert.pem --format yaml > postgres-sealed-secret.yaml

Apply Sealed Secret

Apply the created sealed secret to the Kubernetes cluster, allowing the Sealed Secrets controller to decrypt it and create a standard Kubernetes secret.

kubectl apply -f postgres-sealed-secret.yaml

Verify Secret

Retrieve and decode the values of the username and password stored in the Kubernetes secret to verify that they have been correctly applied

kubectl get secret ss-postgres-secret --namespace=secrets-demo --template=\{\{.data.username}} | base64 -d

kubectl get secret ss-postgres-secret --namespace=secrets-demo --output jsonpath="{.data.password}" | base64 -d

Hashicorp Vault Secrets Operator

HashiCorp Vault is a secret management tool that allows secure storage, access, and tightly controlled management of secrets. It integrates well with Kubernetes, allowing pods to fetch secrets dynamically.

Install Vault Dev Cluster

Creates a namespace for Vault and install the Vault server in your Kubernetes cluster using Helm, enabling the development mode UI and disabling the injector for demo purposes.

kubectl create namespace vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -n vault \
    --set server.dev.enabled=true \
    --set ui.enabled=true \
    --set injector.enabled=false

Configure Vault Cluster

This script enables the KV secrets engine and creates a secret for PostgreSQL configuration in Vault. It also configures Kubernetes authentication and creates a policy and role for accessing the secrets.

vault-config.sh

#!/bin/sh

cd /tmp
# Enable KV secrets engine
vault secrets enable -path=kv-secrets kv-v2
# Create a secret
vault kv put kv-secrets/postgres/config username="admin" password="S3cr3tPassw0rd"

# Enable and configure Kubernetes authentication
vault auth enable -path kubernetes-auth kubernetes
vault write auth/kubernetes-auth/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"


# Create a policy & role for Kubernetes
tee kv-secrets-policy.hcl <<EOF
path "kv-secrets/data/postgres/config" {
   capabilities = ["read", "list"]
}
EOF

vault policy write kv-secrets-policy kv-secrets-policy.hcl

vault write auth/kubernetes-auth/role/vault-k8s-role \
    bound_service_account_names=vault-sa \
    bound_service_account_namespaces=secrets-demo \
    policies=kv-secrets-policy \
    audience=vault \
    ttl=24h

kubectl cp vault-config.sh vault-0:/tmp/vault-config.sh -n vault
kubectl exec -it vault-0 -n vault -- /bin/sh /tmp/vault-config.sh

Install Vault Secrets Operator

This configuration file defines the connection to the Vault instance for the Vault Secrets Operator.

vault-operator-values.yaml

defaultVaultConnection:
  enabled: true
  address: "http://vault.vault.svc.cluster.local:8200"
  skipTLSVerify: false

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
    -n kube-system --values vault-operator-values.yaml

Deploy a Secret

This YAML file defines the service account and the VaultAuth custom resource for connecting the application to Vault using Kubernetes authentication.

vault-auth.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-sa
  namespace: secrets-demo
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: secrets-demo
spec:
  method: kubernetes
  mount: kubernetes-auth
  kubernetes:
    role: vault-k8s-role
    serviceAccount: vault-sa
    audiences:
      - vault

kubectl apply -f vault-auth.yaml

This YAML file defines a VaultStaticSecret resource that specifies the path in Vault where the secret is stored and how it should be injected into Kubernetes.

vault-postgres-secret.yaml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-postgres-secret
  namespace: secrets-demo
spec:
  type: kv-v2
  mount: kv-secrets
  path: postgres/config

  destination:
    name: vault-postgres-secret
    create: true
  refreshAfter: 30s
  vaultAuthRef: vault-auth

kubectl apply -f vault-postgres-secret.yaml

Verify Kubernetes Secret

Retrieve and decode the stored username and password from the Vault static secret in Kubernetes.

kubectl get secret vault-postgres-secret --namespace=secrets-demo --template=\{\{.data.username}} | base64 -d

kubectl get secret vault-postgres-secret --namespace=secrets-demo --output jsonpath="{.data.password}" | base64 -d

Rotate Vault Secret

Update the secret stored in Vault. The updated values will be reflected in the corresponding Kubernetes secret.

kubectl exec -it vault-0 -n vault -- \
    vault kv put kv-secrets/postgres/config username="admin" password="S3cr3tPassw0rd1234"

kubectl get secret vault-postgres-secret --namespace=secrets-demo --template=\{\{.data.username}} | base64 -d

kubectl get secret vault-postgres-secret --namespace=secrets-demo --output jsonpath="{.data.password}" | base64 -d

With Vault, secrets can be dynamically fetched, and access is tightly controlled.

External Secrets Operator

External Secrets Operator integrates external secret management systems (like AWS Secrets Manager, Azure Key Vault, and HashiCorp Vault) with Kubernetes, allowing secrets to be synced directly into Kubernetes.

Install External Secrets Operator

Installs the External Secrets Operator, which facilitates syncing secrets from external providers like AWS Secrets Manager into Kubernetes. Use Helm to install the operator.

kubectl create namespace external-secrets
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets

Here we will use AWS Secrets Manager Provider as Externally managed secret for the demo.

Create AWS Policy & User

Define the IAM policy for accessing AWS Secrets Manager. The necessary permissions are granted to the user.

external-secrets-policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:ListSecrets",
        "secretsmanager:ListSecretVersionIds",
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": ["*"]
    }
  ]
}

external-secrets-user

Create an IAM user and attach the IAM policies directly. Generate Access Keys.

Create AWS credentials Secret

Create a Kubernetes secret containing the AWS access key and secret access key. This secret is referenced by the External Secrets Operator to access AWS resources.

aws-access-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: secrets-demo
type: Opaque
data:
  AWS_ACCESS_KEY_ID: QUtJQVlQWVlYVDRVVEVQTVBaTE4=
  AWS_SECRET_ACCESS_KEY: NlBUblNmdXlhWG0wWGlSN2ErWTJDK2RFM2t2TTE3bEY5eDNkdWJodQ==

DO NOT CHECKIN this file in Version Control System.

It is recommended to use EKS Service Account rather than AWS credentials.

kubectl apply -f aws-access-secret.yaml

Create secret in AWS Secrets Manager

• Region: eu-central-1
• Secret Name: demo/aws-sm/ext-postgres-secret
• Secret Type: Other type of secret
  
• Plaintext: {"username":"admin","password":"S3cr3tPassw0rd"}
  
• Encryption key: aws/secretsmanager
  

Create Secret Store

Define a SecretStore resource in Kubernetes, which is used by the External Secrets Operator to specify how to connect to an external secret management provider, in this case, AWS Secrets Manager.

secret-store.yaml

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-demo
  namespace: secrets-demo
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-secret
            key: AWS_ACCESS_KEY_ID
          secretAccessKeySecretRef:
            name: aws-secret
            key: AWS_SECRET_ACCESS_KEY

kubectl apply -f secret-store.yaml

Create External Secret

Define an ExternalSecret resource that specifies how to sync the secret from AWS Secrets Manager into a Kubernetes secret.

external-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secrets-demo
  namespace: secrets-demo
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: secretstore-demo
  target:
    name: ext-postgres-secret
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: demo/aws-sm/ext-postgres-secret
      property: username
  - secretKey: password
    remoteRef:
      key: demo/aws-sm/ext-postgres-secret
      property: password

kubectl apply -f external-secret.yaml

This SecretStore resource allows the External Secrets Operator to authenticate with AWS Secrets Manager and retrieve secrets securely from the specified region.

Verify Kubernetes Secret

Retrieve and decode the username and password from the Kubernetes secret created by the External Secrets Operator.

kubectl get secret ext-postgres-secret --namespace=secrets-demo --template=\{\{.data.username}} | base64 -d

kubectl get secret ext-postgres-secret --namespace=secrets-demo --output jsonpath="{.data.password}" | base64 -d

Rotate External Secret

Edit secret value

• Plaintext: {"username":"admin","password":"S3cr3tPassw0rd1234"}

Wait for 1 minute before the new Kubernetes secret value is reflected.

kubectl get secret ext-postgres-secret --namespace=secrets-demo --template=\{\{.data.username}} | base64 -d

kubectl get secret ext-postgres-secret --namespace=secrets-demo --output jsonpath="{.data.password}" | base64 -d

Reloader

Reloader can watch changes in Secret as well as ConfigMap and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, Daemonsets, Statefulsets and Rollouts.

Install Reloader

Adds the Stakater Reloader Helm repository and install the Reloader package under the kube-system namespace.

helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader -n kube-system

Create a Deployment

Define the deployment of a demo pod that uses the Reloader annotations to monitor changes in the associated Kubernetes secrets.

• Annotations: The following annotation is added to the pod to enable Reloader functionality:

annotations:
  reloader.stakater.com/auto: "true"

Defines a Pod resource, which uses the busybox image and references Kubernetes secrets for environment variables.

reloader-deploy.yaml

apiVersion: v1
kind: Pod
metadata:
  name: reloader-demo-pod
  namespace: secrets-demo
  labels:
    app: reloader-demo-pod
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  containers:
  - name: reloader-demo-pod
    image: busybox
    env:
    - name: POSTGRES_USER
      valueFrom:
        secretKeyRef:
          name: ext-postgres-secret
          key: username
    - name: POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: ext-postgres-secret
          key: password
    command: ["/bin/sh", "-c", "sleep 3600"]

kubectl apply -f reloader-deploy.yaml

Verify Pod Secret

Retrieve the name of the newly created pod using a label selector to identify the correct pod. Execute a shell in the pod and print the values of the POSTGRES_USER and POSTGRES_PASSWORD environment variables, verifying that the secrets are correctly injected.

pod_name=`kubectl get pod -n secrets-demo -l app=reloader-demo-pod --output=json | jq -r '.items[0].metadata.name'`
kubectl exec -it $pod_name -n secrets-demo -- /bin/sh -c 'echo $POSTGRES_USER'
kubectl exec -it $pod_name -n secrets-demo -- /bin/sh -c 'echo $POSTGRES_PASSWORD'

Rotate External Secret

Edit secret value

• Plaintext: {"username":"admin","password":"S3cr3tPassw0rd123456"}

A one-minute wait is recommended to allow the changes in the external secret to propagate and be reflected in the Kubernetes secret.

pod_name=`kubectl get pod -n secrets-demo -l app=reloader-demo-pod --output=json | jq -r '.items[0].metadata.name'`
kubectl exec -it $pod_name -n secrets-demo -- /bin/sh -c 'echo $POSTGRES_USER'
kubectl exec -it $pod_name -n secrets-demo -- /bin/sh -c 'echo $POSTGRES_PASSWORD'

Managing secrets securely in Kubernetes is critical for maintaining application security. Tools like Sealed Secrets, Vault Secrets, and External Secrets provide powerful ways to manage and inject sensitive data into your cluster while minimizing risks. Each tool offers unique features and benefits, making it easier to choose the right one based on your specific use case.

By using these tools, you can ensure that sensitive data remains secure, whether stored in version control, fetched dynamically from a secure vault, or synced from an external secret management service.