IBM Cloud Global

By: Neela Shah (neela@us.ibm.com)

       Todd Johnson (toddjohn@us.ibm.com) 

Overview

In a world where Cyber Security attacks are in the news everyday, it is extremely critical to ensure that you manage sensitive data such as passwords, API keys and tokens by rotating/changing the values at regular intervals. If you have several applications running in your Enterprise across multiple Kubernetes clusters, you would need to change the sensitive data for each of your applications in every Kubernetes cluster.

Wish there was a central place where you could securely store and change the sensitive data such that all your applications using that sensitive data would automatically get the updates? Your wish has been granted! Secrets Manager lets you create and centrally manage secrets that are used in IBM Cloud services or your own applications!

Secrets Manager and External Secrets Operator

Secrets Manager lets you centralize the secret management, monitor their activity and encrypt them at rest. With Secrets Manager, you can centralize and secure the secrets that are used by the apps that run in your Kubernetes clusters. You can manage the secrets centrally via Secrets Manager, which allows you to rotate the secret when it’s time to do so. 

In addition to storing the secrets securely, you also want to be able to use the secrets from Secrets Manager in your application that is running inside a Redhat OpenShift Kubernetes Cluster (ROKS) in IBM Cloud. In this blog, we will talk about a tool called External Secrets Operator. This package configures the association between Secrets Manager instance and your cluster secrets by creating ExternalSecrets objects that are synchronized to Kubernetes secrets for your application.

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information from external secret management systems and automatically injects the secret values into a Kubernetes Secret.

This blog will cover these concepts and their relationship to each other in more detail using an end to end example in the context of using these with an application that is running inside an IBM Kubernetes Service (IKS) or Redhat OpenShift Kubernetes Cluster (ROKS) in IBM Cloud.

End to End Example

To get started, make sure you have the following pre-requisites completed:

• Create an IKS or ROKS in IBM Cloud in us-east and make sure the cluster is in Normal state. In this example, my cluster in us-east is called `sm-cluster`

• Create a Secrets Manager instance in us-east called `my-app-secrets-manager` and make sure it is provisioned successfully.

• Make sure you have the Secrets Manager CLI plugin installed. It can be installed using `ibmcloud plugin install secrets-manager`

These are the steps that we will follow to accomplish our goals. Each step is explained in more detail below.

1. Prepare your environment

2. Configure your Secrets Manager instance

3. Install the External Secrets Operator (ESO) on your IKS/ROKS cluster

4. Create the necessary ESO resources

5. Create the application pod that will use the secret from Secrets manager

Lets walk through each step in detail.

Prepare your environment

Now that you have a ROKS cluster and a Secrets Manager instance, you need to configure permissions so that you can run operations against both services. You set up an access environment by creating a service ID and an IBM Cloud API key. Run the following commands - 

Note: For the commands, make sure you use the backquote/backtick (`) and not a single quote.

Login to IBM Cloud

ibmcloud login

Create a service ID and set it as an environment variable

export SERVICE_ID=`ibmcloud iam service-id-create k8s-secrets-eso --description "A service ID for ESO integration" --output json | jq -r ".id"`; echo $SERVICE_ID

Assign the service ID permissions to read secrets from Secrets Manager. This will provide the External Secrets Controller the necessary permission to read secrets from Secrets Manager and populate them in a Kubernetes cluster

ibmcloud iam service-policy-create $SERVICE_ID --roles "SecretsReader" --service-name secrets-manager

Create an IBM Cloud API key for your service ID

export IBM_CLOUD_API_KEY=`ibmcloud iam service-api-key-create kubernetes-secrets-tutorial $SERVICE_ID --description "An API key for ESO integration." --output json | jq -r ".apikey"`

Configure your Secrets Manager instance

Export an environment variable with your unique Secrets Manager API endpoint URL

export SECRETS_MANAGER_URL=`ibmcloud resource service-instance my-app-secrets-manager --output json | jq -r '.[].dashboard_url | .[0:-3]'`; echo $SECRETS_MANAGER_URL

Create a secret group for your instance. Secret groups are a way to organize and control who on your team has access to specific secrets in your instance. To create a secret group from the IBM Cloud CLI, you use the ibmcloud secrets-manager secret-group-create command. Run the following command to create a secret group and store its ID as an environment variable.

export SECRET_GROUP_ID=`ibmcloud secrets-manager secret-group-create --name my-app-secret-group --description "Read and write to my app" --output json --service-url $SECRETS_MANAGER_URL | jq -r '.id'`; echo $SECRET_GROUP_ID

Create your app secret in your Secrets Manager instance. Secrets are application-specific and can vary based on the individual app or service that requires them. A secret might consist of a username, password, API key, or any other type of credential.

Secrets Manager supports various types of secrets that you can create and manage in the service. For example, if you need to manage an API key for an app that is protected by IBM Cloud IAM authentication, you can create an IAM credential. Or, if you need to manage a secret that can hold any type of structured or unstructured data, you can create an arbitrary secret.

In our example, we create a secret type of username and password. Run the following command to create the secret and store its ID as an environment variable.

export SECRET_ID=`ibmcloud secrets-manager secret-create --secret-type=username_password --secret-name secret_username_password --username-password-username my-app-user --username-password-password pwd4-my-app --secret-labels "my-app secret" --secret-group-id $SECRET_GROUP_ID --output json --service-url $SECRETS_MANAGER_URL | jq -r '.id'`; echo $SECRET_ID

Install the External Secrets Operator (ESO) on your ROKS cluster

NOTE: If you are using a ROKS cluster for this example, you can install the External Secrets Operator from the Openshift Operator Hub. See Installing with OLM for details. In our example, we are using IKS cluster so we will follow the instructions below using helm.

First, add external-secrets resources to your cluster by installing the official Helm chart. For more installation options, check out the getting started guide.

helm repo add external-secrets https://charts.external-secrets.io

Configure authentication between External Secrets Operator and Secrets Manager using the service id api key we created earlier to authenticate

kubectl -n default create secret generic secret-api-key --from-literal=apikey=$IBM_CLOUD_API_KEY

Install the External secrets operator helm chart

helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace --set installCRDs=true

Create the necessary ESO resources

Create the SecretStore using this yaml (replace the values in < > and remove the <---— comment)-

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-app-secretstore
spec:
  provider:
    ibm:
      serviceUrl: <SECRETS_MANAGER_URL>  <----- Public endpoint from Secrets Manager, exported as SECRETS_MANAGER_URL
      auth:
        secretRef:
          secretApiKeySecretRef:
            name: secret-api-key
            key: apikey

Create the ExternalSecret resource using this yaml (replace the values in < > and remove the <---— comments)-

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-external-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: my-app-secretstore
    kind: SecretStore
  target:
    name: my-app-secure-creds <-- This is the k8s secret that is used by your app
  data:
  - secretKey: username
    remoteRef:
      property: username
      key: username_password/<SECRET_ID> <--- Secret ID from Secrets Manager, exported as SECRET_ID
  - secretKey: password
    remoteRef:
      property: password
      key: username_password/<SECRET_ID> <--- Secret ID from Secrets Manager, exported as SECRET_ID

To validate everything is setup as expected, run the command to see the list of externalsecrets on your cluster-

kubectl get externalsecrets

You should see the following resources-

externalsecret.external-secrets.io/my-app-external-secret             my-secretstore         1h                 SecretSynced        True

secretstore.external-secrets.io/my-app-secretstore     40s   Valid    ReadOnly       True

Now, there should be a kubernetes secret that is created on your cluster based on the secret you had created in Secrets Manger. You can check by listing the secrets as follows-

kubectl get secrets

You should see the kubernetes secret that was created -

my-app-secure-creds                                Opaque                           2      3m24s

You can see the secret data in the kubernetes secret to make sure it is accurate.

kubectl get secret my-app-secure-creds -o json | jq '.data | map_values(@base64d)'

Create the application pod that will use the secret from Secrets manager

Lets create a simple busybox app pod so we can demonstrate the username and password access from an application. We used the following yaml to create the application pod.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: default
spec:
  containers:
  - image: busybox:1.35.0
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    env:
      - name: APP_USERNAME
        valueFrom:
          secretKeyRef:
            name: my-app-secure-creds
            key: username
      - name: APP_PWD
        valueFrom:
          secretKeyRef:
            name: my-app-secure-creds
            key: password
  restartPolicy: Always

You can check the username and password values that the application is seeing via running

kubectl exec -it my-app -- sh

To list the username and password -

/ # echo $APP_USERNAME
my-app-user
/ # echo $APP_PWD
pwd4-my-app
/ # 

Now that the secrets are sync’ed, you can change the secret in Secrets Manager and all the clusters/apps that are setup to use the Secrets Manager secret in this manner will be updated automatically for you. This allows you to have a central place to manage your secrets for your applications.

In our example, I changed the password in my Secrets Manager secret to “new-pwd-4-app” and that got automatically synchronized (based on the refresh interval) into the Kubernetes secret on my cluster and the app pod got updated as below-

kubectl exec -it my-app -- sh                     

/ # echo $APP_PWD

new-pwd-4-app

Conclusion

The full potential, power and usefulness of using Secrets Manager in conjunction with External Secrets operator is a game changer once you have implemented this configuration in your environment!

There are several other features Secrets Manager that we have not covered in detail in this blog like configuring event notifications when secrets are changed or alerts when its time to change the secrets, etc. Since managing secrets centralized with Secrets Manager, these additional features can be the icing on the cake in how you manage sensitive data used in kubernetes applications in your enterprise.

Contact Us

Neela Shah - neela@us.ibm.com

Todd Johnson - toddjohn@us.ibm.com