Power Virtual Server

 View Only

Securing Red Hat OpenShift on IBM Power Virtual Server

By Archana Prabhakar posted 3 days ago

  

Co-author: @JULIE MATHEW

In this digital world, it becomes extremely important to enforce and implement an additional security layer to safeguard our web-based applications on public cloud from being vulnerable to malicious dynamic web attacks. One way to achieve this is by securing the web applications with CA-signed certificates instead of self-signed certificates that are less secure.

This blog shows how to generate custom certificate authority (CA) signed certificates and replace them on a Red Hat® OpenShift® Container Platform 4.x cluster deployed on an IBM® Power Systems™ Virtual Server instance.

It is intended for anyone who deploys OpenShift Container Platform 4.x clusters on Power System Virtual Server. This blog simplifies the procedure of a common use case by replacing the default self-signed certificates with custom CA-signed certificates, which is widely followed by many of the enterprise customers and organizations to strengthen the security of cloud-based deployments.

Prerequisites

Readers of this blog should have basic knowledge about the usage of IBM Cloud, OpenShift Container Platform 4.x and openssl CLI commands. Administrator privilege to the OpenShift Container Platform 4.x cluster is required to replace the custom certificates.

Estimated time

If the CA-signed certificates are already available, then the replacing procedure and post validation checks would take around 10 to 20 minutes.

If the CA-signed certificates have to be generated, followed by the replacement and post validation checks, then the entire procedure would take around 20 to 30 minutes.

Adding custom CA-signed certificates to the OpenShift cluster

If custom CA-signed certificates are already generated using other methods or tools, you can proceed with Case 1. If not generated already, proceed with Case 2.

Case 1: Custom CA-signed certificates are already generated

If the custom CA-signed certificates are already generated, perform the following steps:

  1. Log in to the helper node (bastion node).

  2. Create a new directory.

    mkdir ~/certificates
cd ~/certificates

  3. Copy the custom CA-signed certificates and the root CA certificates to this directory.

  4. Proceed to step 9 under Case 2 and follow the next set of the steps in the same order.

Case 2: Custom CA-signed certificates need to be generated

If the custom CA-signed certificates have to be generated, perform the following steps:

  1. Log in to the helper node (bastion node).

  2. Create a new directory.

    mkdir ~/certificates
cd ~/certificates

  3. Generate a root CA key using the openssl command.

    openssl genrsa -out OCPRootCA.key 2048
  4. Generate a root CA certificate using the root CA key generated in step 3. 
    openssl req -x509 -new -nodes -key OCPRootCA.key -sha256 -days 1024 -out OCPRootCA.pem

    Note: Specify the helper node's hostname when prompted for a common name.

  5. Generate the client key for the helper node. 
    openssl genrsa -out OCPClient1.key 2048
  6. Create a config file at the /etc/ssl path with the cluster details including the common name and the subject alt name for the wildcard applications with the *.app. 
    touch /etc/ssl/openssl_ingress.cnf

    Refer to the following example of the contents of the /etc/ssl/openssl_ingress.cnf file.

    [ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no
default_md = sha256

[ req_distinguished_name ]
C = IN
ST = KAR
L = BLR
O = IBM Cloud PowerVS
OU = OCP
CN = *.apps.test-arc-crt45.ibm.com
emailAddress = xxx@in.ibm.com

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = 169.48.19.214
DNS.2 = test-arc-crt45-bastion.ibm.com
DNS.3 = *.apps.test-arc-crt45.ibm.com
  7. Create a certificate signing request (CSR) file. 
    openssl req -new -key OCPClient1.key -out OCPClient1.csr -reqexts req_ext -config /etc/ssl/openssl_ingress.cnf
  8. Generate a custom CA-signed server certificate using the client CSR generated in step 7. 
    openssl x509 -req -in OCPClient1.csr -CA OCPRootCA.pem -CAkey OCPRootCA.key -CAcreateserial -out OCPClient1.pem -days 365 -sha256
  9. Update the OS trust store with the root CA key and root CA certificate. 
    cp OCPRootCA.pem  /etc/pki/ca-trust/source/anchors
cp OCPRootCA.key /etc/pki/CA/private
update-ca-trust extract

    If the /etc/pki/CA/private directory does not exist, create it.

Replacing the default ingress certificate with the custom CA-signed certificate

Perform the following steps to replace the default ingress certificate with the custom CA-signed certificate on an OpenShift Container Platform 4.x cluster:

  1. Create a config map with the custom root CA certificate. 
    oc create configmap custom-ca --from-file=ca-bundle.crt=/root/certificates/OCPRootCA.pem -n openshift-config
  2. Verify that the config map is created in the openshift-config namespace. 
    oc get configmap --all-namespaces | grep custom-ca
openshift-config           custom-ca       1      33s
  3. Patch the cluster with the new custom CA config created. 
    oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
  4. Create a Transport Layer Security (TLS) ingress secret with the custom CA-signed server certificate and server key in the openshift-ingress namespace. 
    oc create secret tls ingress-secret --cert=/root/certificates/OCPClient1.pem --key=/root/certificates/OCPClient1.key -n openshift-ingress
  5. Patch the ingress controller operator with the new ingress secret. 
    oc patch ingresscontroller.operator default --type=merge -p  '{"spec":{"defaultCertificate": {"name": "ingress-secret"}}}' -n openshift-ingress-operator

    The following cluster operators get restarted. Wait for some time, until all of them are running.

    dns
Kube-apiserver
Kube-controller-manager
Kube-scheduler
Openshift-apiserver
Network
Console
Authentication
  6. Run the oc get co command to monitor the status of the cluster operators.
  7. Check if all the nodes are running and pods are healthy. 
    oc get pods --all-namespaces | grep -vi completed | grep -vi running
NAMESPACE
NAME
READY   STATUS      RESTARTS   AGE

    Run the oc login command and ensure that your login is successful.

    oc login
Authentication required for https://api.test-arc-crt45.ibm.com:6443 (openshift)
Username: kubeadmin
Password:
Login successful.

You have access to 59 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
  8. Add the custom certificate (OCPClient1.pem) into the client system's certificate trust store as follows. (The following steps are for a Mac client system.)
    1. Download the custom certificate to the local system (Mac).
    2. Launch the Keychain access app from the path, Applications/Utilities/Keychain Access.app.
    3. Drag the custom certificate to the Certificates folder of the Keychain access app.
    4. Double-click the certificate, expand the Trust menu, and from the When using this certificate drop-down menu, select the Always Trust option.
  9. Launch the UI link of the OpenShift Container Platform cluster. Ensure that you find the lock symbol against the URL in the browser. Click the lock symbol and select Show certificate to view the content and the trusted status. Then, click OK to close the menu. Enter the username and the password on the login page. Now you should be able to login successfully.

Summary

There are multiple ways to generate custom CA-signed certificates for IBM Cloud-based deployments. As a next step, you can use the Let's Encrypt tool to order CA-signed certificates for your OpenShift Container Platform 4.x cluster or the custom web-based applications that you host on the cluster. If you already have custom certificates for IBM Cloud deployments, you can configure a certificate manager to import and manage all your custom certificates.

0 comments
1 view

Permalink