This article was originally posted to Medium by Gaurav Bankar and has been updated.
Hi Folks, I am introducing a topic related to OpenShift Container Platform on Power security.
Let’s dive into the topic!
Encrypting etcd
By default, etcd data is not encrypted in the OpenShift Container Platform. You can enable etcd encryption for your cluster to provide an additional layer of security.
When you enable etcd encryption, encryption keys are created and the following OpenShift and Kubernetes resources are encrypted:
- Secrets
- Config maps
- Routes
- OAuth access tokens
- OAuth authorize tokens
To further protect etcd, the encryption keys are rotated on a weekly basis. You must have these keys to restore from an etcd backup.
Enabling etcd encryption
You must be logged into the cluster with clusteradmin credentials.
- Modify the APIServer object
oc edit apiserver
- Set the encryption field type to
aescbc or aesgcm:
spec:
encryption:
type: aescbc
The aescbc type means that AES-CBC with PKCS#7 padding and a 32 byte key are used to perform the encryption. The aesgcmtype is AES-GCM.
- Save the changes and the cluster starts the reconciliation process.
The encryption process starts. It can take 20 minutes or longer for this process to complete, depending on your cluster size.
- Verify that the etcd encryption was successful.
a. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully decrypted:
oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
The output shows EncryptionCompleted upon successful encryption:
EncryptionCompleted
All resources encrypted: routes.route.openshift.io
b. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully encrypted:
oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
The output shows EncryptionCompleted upon successful encryption:
EncryptionCompleted
All resources encrypted: secrets, configmaps
c. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully encrypted:
oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
The output shows EncryptionCompleted upon successful encryption:
EncryptionCompleted
All resources encrypted: oauthaccesstokens.oauth.openshift.io, oauthauthorizetokens.oauth.openshift.io
Note: If the output shows EncryptionInProgress, encryption is still in progress. Wait for a few minutes and try again.
Disabling etcd encryption
- Modify the APIServer object:
oc edit apiserver
- Set the encryption field type to identity:
spec:
encryption:
type: identity
The identity type is the default value and means that no encryption is performed.
- Save the file to apply the changes.
The decryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of your cluster.
- Verify that etcd decryption was successful.
a. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully decrypted:
oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}
The output shows DecryptionCompleted upon successful decryption:
DecryptionCompleted
Encryption mode set to identity and everything is decrypted
b. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully decrypted:
oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
The output shows DecryptionCompleted upon successful decryption:
DecryptionCompleted
Encryption mode set to identity and everything is decrypted
c. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully decrypted:
oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
The output shows DecryptionCompleted upon successful decryption:
DecryptionCompleted
Encryption mode set to identity and everything is decrypted
Force encryption key rotation for etcd datastore
This step is for verification of how key rotation will work for etcd datastore.
- Enable encryption. (Encryption is already enabled.
To check secret before patching, below are the secrets.
# oc get secret -n openshift-config-managed | grep enc
encryption-config-openshift-apiserver Opaque 1 20h
encryption-config-openshift-kube-apiserver Opaque 1 20h
encryption-config-openshift-oauth-apiserver Opaque 1 22h
encryption-key-openshift-apiserver-1 Opaque 1 22h
encryption-key-openshift-apiserver-3 Opaque 1 20h
encryption-key-openshift-apiserver-4 Opaque 1 20h
encryption-key-openshift-apiserver-5 Opaque 1 5h23m
encryption-key-openshift-kube-apiserver-1 Opaque 1 22h
encryption-key-openshift-kube-apiserver-3 Opaque 1 20h
encryption-key-openshift-kube-apiserver-4 Opaque 1 5h23m
encryption-key-openshift-oauth-apiserver-1 Opaque 1 22h
# oc get po -n openshift-apiserver -l apiserver --show-labelsNAME READY STATUS RESTARTS AGE LABELSapiserver-7fb8886978-j94hj 2/2 Running 0 5h14m apiserver=true,app=openshift-apiserver-a,openshift-apiserver-anti-affinity=true,pod-template-hash=7fb8886978,revision=13apiserver-7fb8886978-kj7mv 2/2 Running 0 5h12m apiserver=true,app=openshift-apiserver-a,openshift-apiserver-anti-affinity=true,pod-template-hash=7fb8886978,revision=13apiserver-7fb8886978-rv9x2 2/2 Running 0 5h14m apiserver=true,app=openshift-apiserver-a,openshift-apiserver-anti-affinity=true,pod-template-hash=7fb8886978,revision=13
- Before patching we can create secret file
Check existing and newly created resources are ALL encrypted in etcd. Get openshift etcd pods select one pod and log in it:
# ETCD_POD=$(oc get po -n openshift-etcd -l app=etcd | grep ^etcd- | head -1 | awk '{print $1}')
Command for login into the pod:
oc rsh -n openshift-etcd $ETCD_POD
Defaulted container "etcdctl" out of: etcdctl, etcd, etcd-metrics, etcd-health-monitor, etcd-readyz, setup (init), etcd-ensure-env-vars (init), etcd-resources-copy (init)
- Generate secret file with keys:
sh-4.4# NUMBER=4
sh-4.4# for i in /kubernetes.io/secrets /kubernetes.io/configmaps /openshift.io/routes /openshift.io/oauth/accesstokens
do
FILE=/tmp/"${i##*/}"-values-$NUMBER.txt
etcdctl get $i --prefix -w fields > "$FILE-with-key"
etcdctl get $i --prefix -w fields | grep -e '"Value"' > $FILE
echo "${i##*/} items: `cat $FILE | wc -l` prefix: `grep k8s:enc:aescbc:v1:$NUMBER $FILE | wc -l`"
done
These resource values should all have the k8s:enc:aescbc:v1:$NUMBER prefix: Get below output, item number == prefix number, means all resources are
encrypted:
secrets items: 988 prefix: 988
configmaps items: 314 prefix: 314
routes items: 9 prefix: 9
accesstokens items: 2 prefix: 2secrets items: 917 prefix: 917
configmaps items: 549 prefix: 549
routes items: 8 prefix: 0
accesstokens items: 16 prefix: 0
Below are the Files which are created for secret by using above scripts.(Before patching )
sh-4.4# ls
configmaps-values-4.txt secrets-values-4.txt
accesstokens-values-4.txt configmaps-values-4.txt-with-key routes-values-4.txt secrets-values-4.txt-with-key
accesstokens-values-4.txt-with-key routes-values-4.txt-with-key secrets-values-5.txt
Before patching value of key:
cat secrets-values-4.txtkey name is: "Key" : "/kubernetes.io/secrets/default/builder-dockercfg-mlf9c""Value" : "k8s:enc:aescbc:v1:4:\x83\xa5\xba\xeev\x1b\xc5gO\x15▒~T\r#▒~I\x9e\x16a\x05\xf1\xddP\x97\xfaW:ice\xbe0E\xd4\u007f|\x92\xf4N\xfe\xd4d\x1a*\xe
- Force key rotation
To rotate key forcefully we can use below steps:
Force key rotation by adding below:
# oc patch openshiftapiserver cluster --type merge -p "
spec:
unsupportedConfigOverrides:
encryption:
reason: force OAS rotation `date`
"
openshiftapiserver.operator.openshift.io/cluster patched# oc patch kubeapiserver cluster --type merge -p "
spec:
unsupportedConfigOverrides:
encryption:
reason: force KAS rotation `date`
"
kubeapiserver.operator.openshift.io/cluster patched
- Verification of secret after patching:
oc get secret encryption-key-openshift-kube-apiserver-5 encryption-key-openshift-apiserver-5 -n openshift-config-managed -o yaml | grep migrated encryption.apiserver.operator.openshift.io/migrated-resources: '{"resources":[{"Group":"","Resource":"configmaps"},{"Group":"","Resource":"secrets"}]}'
encryption.apiserver.operator.openshift.io/migrated-timestamp: "2022-09-21T14:08:16Z"
encryption.apiserver.operator.openshift.io/migrated-resources: '{"resources":[{"Group":"route.openshift.io","Resource":"routes"}]}'
encryption.apiserver.operator.openshift.io/migrated-timestamp: "2022-09-21T08:10:02Z"
- Verify new secret after patching:
oc get secret -n openshift-config-managed | grep enc
encryption-config-openshift-apiserver Opaque 1 22h
encryption-config-openshift-kube-apiserver Opaque 1 22h
encryption-config-openshift-oauth-apiserver Opaque 1 24h
encryption-key-openshift-apiserver-1 Opaque 1 24h
encryption-key-openshift-apiserver-3 Opaque 1 22h
encryption-key-openshift-apiserver-4 Opaque 1 22h
encryption-key-openshift-apiserver-5 Opaque 1 7h9m
encryption-key-openshift-apiserver-6 Opaque 1 82m
encryption-key-openshift-kube-apiserver-1 Opaque 1 24h
encryption-key-openshift-kube-apiserver-3 Opaque 1 22h
encryption-key-openshift-kube-apiserver-4 Opaque 1 7h8m
encryption-key-openshift-kube-apiserver-5 Opaque 1 82m
encryption-key-openshift-oauth-apiserver-1 Opaque 1 24h
- To verify value of key after Force key rotation and check existing and newly created resources are ALL encrypted in etcd.
# ETCD_POD=$(oc get po -n openshift-etcd -l app=etcd | grep ^etcd- | head -1 | awk '{print $1}')
- Login to the Pod
oc rsh -n openshift-etcd $ETCD_PODDefaulted container "etcdctl" out of: etcdctl, etcd, etcd-metrics, etcd-health-monitor, etcd-readyz, setup (init), etcd-ensure-env-vars (init), etcd-res ources-copy (init)sh-4.4# NUMBER=5
- To generate secret file with keys:
sh-4.4# for i in /kubernetes.io/secrets /kubernetes.io/configmaps /openshift.io/routes /openshift.io/oauth/accesstokens
do
FILE=/tmp/"${i##*/}"-values-$NUMBER.txt
etcdctl get $i --prefix -w fields > "$FILE-with-key"
etcdctl get $i --prefix -w fields | grep -e '"Value"' > $FILE
echo "${i##*/} items: `cat $FILE | wc -l` prefix: `grep k8s:enc:aescbc:v1:$NUMBER $FILE | wc -l`"
done
These resource values should all have the k8s:enc:aescbc:v1:$NUMBER prefix:
- Get below output, item number == prefix number, means all resources are encrypted:
secrets items: 988 prefix: 988
configmaps items: 314 prefix: 314
routes items: 9 prefix: 9
accesstokens items: 2 prefix: 2secrets items: 917 prefix: 917
configmaps items: 549 prefix: 549
routes items: 8 prefix: 0
accesstokens items: 16 prefix: 0
- Below are the Files which are created for secret by using above scripts. (Before patching i.e. prefix with 4 and after patching prefix with 5 )
sh-4.4# lsconfigmaps-values-4.txt secrets-values-4.txt accesstokens-values-4.txt configmaps-values-4.txt-with-key routes-values-4.txt secrets-values-4.txt-with-keyaccesstokens-values-4.txt-with-key configmaps-values-5.txt routes-values-4.txt-with-key secrets-values-5.txtaccesstokens-values-5.txt configmaps-values-5.txt-with-key routes-values-5.txt secrets-values-5.txt-with-keyaccesstokens-values-5.txt-with-key ks-script-3y_ml57e routes-values-5.txt-with-key
- Check the values of keys after patching and Here we have compared values of secret file of secrets-values-4.txt and secrets-values-5.txt and the key name is:
"Key" : "/kubernetes.io/secrets/default/builder-dockercfg-mlf9c""Value" : "k8s:enc:aescbc:v1:5:\xa4E\xd7oA▒~N\x9e+\xe4\x83\n\x83\xb5_\xe5\xcd\x10\x9a\x06\x82T\xa2\xfc\x14\xa5\x17\xb5\x05\xd9o\x83b<H\xe8\xc6\xcb<▒~D\"
So, if we see version as 5 in this file and values are different in both the files which is expected.
As you can see we have covered important points related to Encrypting etcd data, we can conclude this blogpost for now.
Thanks for reading! I hope you found this tutorial helpful :)
References
- https://docs.openshift.com/container-platform/4.11/security/encrypting-etcd.html
- https://docs.openshift.com/container-platform/4.14/security/encrypting-etcd.html