This post looks at how two open source technologies can be used to externalize storage and management of secrets from a Kubernetes cluster.
- HashiCorp Vault: A popular open source secrets store.
-
Secrets Store CSI Driver: Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.
Why externalize secrets from Kubernetes?
The default secrets storage in Kubernetes probably allows for more unmanaged access to those secrets than you would like.
When you create a Secret in Kubernetes, the secret value is Base64 encoded and written to the cluster's etcd store.
Anyone with suitably privileged access to the cluster can view the secret payload and decode those values. Anyone with API access can retrieve or modify a Secret, as can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.
Furthermore, by default the etcd storage (data at rest) is not encrypted, making physical theft of secrets an additional risk.
The Secrets Store CSI driver provides a standard way to declare secrets which it then sources from a secrets store (e.g. HashiCorp Vault), mounts into pods as an ephemeral tmpfs volume and mounts the secret data items into the container's file system at runtime.
If you can exec* into the Pod, you can read the secret. Nevertheless, numerous security posture improvements result, including
- No way to access the secret value in the cluster without access to the Pod.
- No way to change the secret value in the cluster (can only be changed in the secret store).
- Secrets are injected to containers at runtime. No persistent storage in the cluster. Secrets are ephemeral.
- Delegation of secret management to a system which specialises in that function.
- Secure role and policy based access to secrets in the secret store scoped to specific workloads.
- Simplified secret rotation (not covered in this post).
* Can be completely mitigated using distroless container images.
Setup
The Secrets Store CSI Driver should be installed in the Kubernetes cluster, e.g. with helm
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo update
helm install csi secrets-store-csi-driver/secrets-store-csi-driver
Vault can be installed in Kubernetes or elsewhere.
The Vault CSI provider should be installed in the Kubernetes cluster, e.g. with helm
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault --values <as required> \
--set "server.enabled=false" \
--set "injector.enabled=false" \
--set "csi.enabled=true"
If Vault is to be run in the Kubernetes cluster, set server.enabled=true.
Vault configuration
Write the secret payload to Vault
vault kv put secret/app username="admin" password="p4ssw0rd"
Create a policy and role which will authorize and authenticate the interaction between Vault and Kubernetes
vault policy write mycsipolicy - <<EOF
path "secret/data/app" {
capabilities = ["read"]
}
EOF
vault write auth/kubernetes/role/mycsirole \
bound_service_account_names=mycsiserviceaccount \
bound_service_account_namespaces=default \
policies=mycsipolicy \
ttl=20m
CSI SecretProviderClass
This declares the structure of the secret data which will be read from the external secret store, and which provider will do so.
- Declares the secret payload structure
- Declares the CSI Provider to be vault and the Vault server address
- Declares the Vault role which will authenticate the Deployment's interaction between Vault and Kubernetes
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: mycsispc
spec:
provider: vault
parameters:
roleName: mycsirole
vaultAddress: http[s]://<vault-server>:8200
objects: |
- objectName: "myUsername"
secretPath: "secret/data/app"
secretKey: "username"
- objectName: "myPassword"
secretPath: "secret/data/app"
secretKey: "password"
App deployment
This is an outline of the deployment for an app which obtains secret data via the Secrets Store CSI driver.
- The declared serviceAccount will be used by Vault in Kubernetes authentication
- The serviceAccount will be bound in Vault to a Vault role and policy
- The secret(s) are declared at a mount point
- The volume spec references the (general) CSI driver and the (secret-specific) secretProviderClass
apiVersion: apps/v1
kind: Deployment
metadata:
name: mycsiapp
labels:
app: mycsiapp
spec:
selector:
matchLabels:
app: mycsiapp
template:
metadata:
labels:
app: mycsiapp
spec:
serviceAccountName: mycsiserviceaccount
containers:
- image: <your image>
name: csiapp
volumeMounts:
- name: appsecrets
mountPath: "/mnt/csi"
readOnly: true
volumes:
- name: appsecrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: mycsispc
Deploy the application components
kubectl create serviceaccount mycsiserviceaccount
kubectl apply -f <service provider class>
kubectl apply -f <app deployment>
The result
The secret payload objects are mounted to the container at runtime
$ kubectl exec deploy/mycsiapp -- ls -l /mnt/csi
total 0
lrwxrwxrwx 1 root root 17 Oct 4 12:20 myPassword -> ..data/myPassword
lrwxrwxrwx 1 root root 17 Oct 4 12:20 myUsername -> ..data/myUsername
$
$ kubectl exec deploy/mycsiapp -- cat /mnt/csi/myPassword
p4ssw0rd%
PS
Note that while this post demonstrates externalization of secrets to HashiCorp Vault, the Secrets Store CSI driver integrates in exactly the same way with several other secrets store backends. See Supported Providers.
References