Overview
IBM Cloud Pak System accelerates your implementation of on-premises Kubernetes platforms. It comes with support for automated deployment and configuration of Red Hat OpenShift Container Platform (OCP). This makes it the perfect platform for on-premises deployment of IBM Cloud Paks and Red Hat OpenShift clusters!
This tutorial focuses on the deployment of Red Hat OpenShift Container Platform 4.6 on IBM Cloud Pak System V2.3.3.3. For details on version 4.4, refer to the IBM Developer article Deploying Red Hat OpenShift 4.4 on IBM Cloud Pak System.
For Red Hat OpenShift, it is important to know that there are several different offerings available:
A fully managed public cloud offering for quickly deploying applications.
- OpenShift Hosted Services
OpenShift clusters hosted on IBM Cloud, Amazon Web Services (AWS), and Azure.
- OpenShift Container Platform (OCP)
An enterprise OpenShift cluster deployed on your own on-premises infrastructure (OpenShift Container Platform was previously called OpenShift Enterprise, but the name was changed with the release of version 3.3.).
A more detailed comparison of these offerings can be found on the OpenShift website. As IBM Cloud Pak System is an on-premises appliance, it only provides support for the OpenShift Container Platform offering. In this tutorial, you will learn how to deploy OpenShift Container Platform on IBM Cloud Pak System. Steps are written assuming that the IBM Cloud Pak System is at 2.3.3.3 firmware, and does not have direct access to the internet.
Prerequisites
Before you deploy your first OpenShift 4.6 cluster on IBM Cloud Pak System, few prerequisites need to be in place. For a good starting point on these prerequisites, see IBM Knowledge Center:
- IBM Cloud Pak System 2.3.3.3 Intel based IBM Cloud Pak System models W2500, W3500, W3550, and W4600 are supported. There is currently no support for the Power based IBM Cloud Pak System model W3700.
- IBM OS image for Red Hat Linux Systems (RHEL 7.8 X64) Version 7.8 Scenarios using a custom OS image are also supported, as long as it is Red Hat Enterprise Linux (RHEL) 7.8 or higher.
- IBM OS image for Red Hat Enterprise Linux CoreOS Version 4.6.0 This is included with accelerator bundle for OpenShift Container Platform 4.6.0.0.
- Docker Accelerator 1.0.14.0 Docker registry is required for accessing OpenShift Container Platform images. Obtain Docker registry accelerator 1.0.14.0 from IBM FixCentral.
- OpenShift Container Platform Accelerator 4.6.0.0 OpenShift Container Platform accelerator is used for deploying OpenShift Container Platform cluster in IBM Cloud Pak System. Get OpenShift Container Platform accelerator 4.6.0.0 from IBM FixCentral.
- Red Hat Satellite Server 6 shared service deployed The shared service should be connected to an existing Red Hat Satellite Server (RHSS), or to RHSS deployed on IBM Cloud Pak System with access to repositories rhel-7-server-rpms & rhel-7-server-extras-rpms.
Note that IBM Cloud Pak System comes with Red Hat subscriptions for RHEL and RHSS.
- Active subscription with Red Hat for the OpenShift Container Platform Unlike the Red Hat subscription for RHEL and RHSS, the OpenShift Container Platform (OCP) subscription is not included with IBM Cloud Pak System. Please refer to OpenShift Container Platform accelerator V4.3.1.0 recipe for detailed steps.
- Sufficient compute, memory and storage resources on IBM Cloud Pak System A single OpenShift Container Platform cluster requires at least 28 virtual CPUs, 112 GB of RAM, and 1202 GB of storage.
For detailed prerequisite steps to configure Red Hat Satellite Server and Private Docker Registry on IBM Cloud Pak System, refer to OpenShift Container Platform accelerator V4.3.1.0 recipe.
Step-by-step
- Load and verify required artefacts on IBM Cloud Pak System V2.3.3.3
By default, IBM Cloud Pak System 2.3.3.0 includes most of the required content pre-loaded for you.
For the sake of completeness, see the following complete list of everything that you need:
The “IBM Cloud Pak System accelerator bundle for Red Hat Openshift V4.6.0” are not installed by default. To confirm, log in to IBM Cloud Pak System and go to System > Storehouse Browser. If you do not see an entry for /admin/files/RedHatOpenShift as shown in Figure 1, then it means that the binaries are not installed yet.
Figure 1
Process for loading the binaries has been documented here in the IBM Cloud Pak System Knowledge Center.
./cloudpakimport.sh -h 9.xx.xx.xx -u <IBM Cloud Pak System user> -p <IBM Cloud Pak System user password>
***************************************************
Cloud Pak Accelerator Bundle Import utility
***************************************************
Artifacts from accelerator Bundle will be imported to Cloud Pak System
- Testing connectivity to Cloud Pak System
Upload Cloud Pak binaries to Storehouse
-----------------------------------------------
- Uploading binaries for: Red Hat OpenShift 4.6.1 on IBM Cloud Pak System
- Pre-upload Verification
1) openshift-install-linux-4.3.1.tar.gz: found locally but not on server
2) openshift-client-linux-4.6.1.tar.gz: found locally but not on server
3) ocp4.4.6-x86_64.tgz: found locally but not on server
4) openshift-install-linux-4.4.6.tar.gz: found locally but not on server
5) openshift-install-linux-4.6.1.tar.gz: found locally but not on server
6) openshift-client-linux-4.4.6.tar.gz: found locally but not on server
7) ocp4.3.1-x86_64.tgz: found locally but not on server
8) ocp4.6.1-x86_64.tgz: found locally but not on server
9) ocp4.3.1-x86_64-extra.tar: found locally but not on server
10) jq-linux64: found locally but not on server
11) openshift-client-linux-4.3.1.tar.gz: found locally but not on server
- Uploading openshift-install-linux-4.3.1.tar.gz
- openshift-install-linux-4.3.1.tar.gz uploaded
- Uploading openshift-client-linux-4.6.1.tar.gz
- openshift-client-linux-4.6.1.tar.gz uploaded
- Uploading ocp4.4.6-x86_64.tgz
- ocp4.4.6-x86_64.tgz uploaded
- Uploading openshift-install-linux-4.4.6.tar.gz
- openshift-install-linux-4.4.6.tar.gz uploaded
- Uploading openshift-install-linux-4.6.1.tar.gz
- openshift-install-linux-4.6.1.tar.gz uploaded
- Uploading openshift-client-linux-4.4.6.tar.gz
- openshift-client-linux-4.4.6.tar.gz uploaded
- Uploading ocp4.3.1-x86_64.tgz
- ocp4.3.1-x86_64.tgz uploaded
- Uploading ocp4.6.1-x86_64.tgz
- ocp4.6.1-x86_64.tgz uploaded
- Uploading ocp4.3.1-x86_64-extra.tar
- ocp4.3.1-x86_64-extra.tar uploaded
- Uploading jq-linux64
- jq-linux64 uploaded
- Uploading openshift-client-linux-4.3.1.tar.gz
- openshift-client-linux-4.3.1.tar.gz uploaded
- Post-upload verification
1) openshift-install-linux-4.3.1.tar.gz: verified successfully
2) openshift-client-linux-4.6.1.tar.gz: verified successfully
3) ocp4.4.6-x86_64.tgz: verified successfully
4) openshift-install-linux-4.4.6.tar.gz: verified successfully
5) openshift-install-linux-4.6.1.tar.gz: verified successfully
6) openshift-client-linux-4.4.6.tar.gz: verified successfully
7) ocp4.3.1-x86_64.tgz: verified successfully
8) ocp4.6.1-x86_64.tgz: verified successfully
9) ocp4.3.1-x86_64-extra.tar: verified successfully
10) jq-linux64: verified successfully
11) openshift-client-linux-4.3.1.tar.gz: verified successfully
- All binaries verified successfully.
- Verifying file
Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
- Verifying file
Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
- Verifying file
Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_6_1.ova
- Downloading CLI from Cloud Pak System
- Starting Cloud Pak System CLI
/
Import and Clone Virtual Images
----------------------------------------
- Checking for Virtual Image
Name: IBM OS Image for Red Hat Linux Systems
Version: 3.1.2.0
- Virtual Image is available
- Working with image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
Version: 4.3.0
- Virtual Image will be imported from
IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
- Waiting for import to finish. This would take few minutes.
- Cloning the image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
- Waiting for clone to finish
- Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
- Working with image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
Version: 4.4.3
- Virtual Image will be imported from
IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
- Waiting for import to finish. This would take few minutes.
- Cloning the image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
- Waiting for clone to finish
- Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
- Working with image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
Version: 4.6.1
- Virtual Image will be imported from
IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_6_1.ova
- Waiting for import to finish. This would take few minutes.
- Cloning the image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
- Waiting for clone to finish
- Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_6_1.ova
Import Ptypes
------------------------
- Import ptype for: Red Hat OpenShift 4.6.1 on IBM Cloud Pak System
- Working with ptype
Name: openshift
Version: 4.6.0.0
- Ptype will be imported from /home/byol_openshift/openshift-4.6.0.0.tgz
--/
Confirm that the following files are now visible from the IBM Cloud Pak System. Go to System > Storehouse Browser and confirm that you see what is shown in Figure 2.
Figure 2
New and improved Cloud Pak accelerator bundle import utility is available for OpenShift Container Platform V4.6.0.0 accelerator on IBM Cloud Pak System. It loads images to storehouse, clones the Red Hat Enterprise Linux CoreOS OVA to two variants with different disk size of 16 GB and 120 GB, and makes them available in IBM Cloud Pak System Virtual Images catalog as shown in Figure 3. In addition, it also imports the accelerator into IBM Cloud Pak System.
Figure 3
- Verify and import Pattern Types
The new and improved Cloud Pak accelerator bundle import utility (described in previous step) for IBM Cloud Pak System loads pattern type to IBM Cloud Pak System too. After it is loaded, you should see the OpenShift 4.6.0.0 accelerator in the catalog with status available as shown in Figure 4.
Figure 4
- Explore Red Hat OpenShift 4 cluster accelerator
Navigate to Provision accelerators from IBM Cloud Pak System user interface as shown in Figure 5.
Figure 5
Search for OpenShift Container Platform and click Ready to deploy as shown in Figure 6.
Figure 6
This action opens a page to select Default or Customized cluster.
Select Customized as shown in Figure 7 and click Continue. The Configure deployment page opens.
Figure 7
The first section in this page includes IBM Cloud Pak System deployment options as shown in Figure 8. For example, environment profile, cloud group, and so on. Select the appropriate values such that all prerequisites are available in the chosen cloud group.
Figure 8
The second section in this page contains OS users root and virtuser credentials for helper virtual machines as shown in Figure 9.
Figure 9
The last section in this page includes an optional parameter to generate SSH key, which is used in deployment as shown in Figure 10. If you generate this key, you can use it to connect to helper and OpenShift Container Platform virtual machines via SSH.
Figure 10
Click Configure nodes to open the Configure nodes tab page.
This page shows type of VMs, such as helpers, control planes and worker nodes, which are included in the accelerator and hardware resource configuration for each virtual machine as shown in Figure 11 and Figure 12. You can change the resource configuration of all nodes in a cluster to be deployed. Use slider, as shown in Figure 11, to set the desired value for each resource type of virtual machines.
Figure 11
Note: You cannot change number of masters to be deployed as Red Hat recommends deploying three masters for OpenShift Container Platform 4 clusters.
Figure 12
Click OpenShift options to open the OpenShift options tab page.
This page shows deployment parameters for OpenShift Container Platform accelerator as shown in Figures 13 and 14. Detailed description of parameters can be found here.
OpenShift version
It is the Red Hat OpenShift Container Platform version to deploy.
OpenShift cluster domain name
It is the domain name for Red Hat OpenShift Container Platform cluster. If not specified, then a default cluster domain name is generated.
OpenShift cluster name
It is the Red Hat OpenShift Container Platform cluster name. If not specified, then a default cluster name is generated.
OpenShift image registry name
It is the fully qualified name for the Red Hat OpenShift Container Platform image registry. In case there is no Cloud Pak System Registry available in cloud group or you wish to use a different docker registry for accessing OpenShift Container Platform images, specify registry hostname:port here.
OpenShift image registry username
It is the username needed to access the Red Hat OpenShift Container Platform secure image registry.
OpenShift image registry password
It is the user password that is needed to access the Red Hat OpenShift Container Platform secure image registry.
OpenShift pull-secret
If access to external OpenShift registry is available and you wish to use it for OpenShift Container Platform images, paste contents of the downloaded OpenShift install pull-secret.json file here.
Figure 13
Alternate NFS server name or IP address (for the OpenShift image registry)
It is the alternate NFS server name or IP address of the OpenShift image registry. By default, Primary helper virtual machine is used to configure storage for OpenShift image registry. If this parameter value is specified, then this external NFS server is used for OpenShift image registry instead. Please see this page about configuring NFS server and path.
Alternate NFS server path (for the OpenShift image registry)
It is the alternate NFS server path of the OpenShift image registry. It is used along with the Alternate NFS server name or IP address parameter to configure storage for OpenShift image registry.
Alternate NFS server name or IP address (for the application persistent storage)
It is the alternate NFS server name or IP address of the Application Persistent Storage. By default Primary helper is used to configure persistent storage for workloads to be deployed on OpenShift Container Platform cluster. When value is specified here, it is used for persistent storage for workloads instead.
Alternate NFS server path (for the application persistent storage)
It is the alternate NFS server path of the Application Persistent Storage. It is used along with Alternate NFS server name or IP address parameter to configure persistent storage for workloads.
Figure 14
Click Review and deploy to open the Ready to deploy tab page. This page shows a summary of topology and resource configuration as shown in Figure 15.
Figure 15
Following is a brief description of various nodes included in cluster.
PrimaryHelper and SecondaryHelper
There are two Helper nodes that run on RHEL 7.7. It supports the deployment of OpenShift cluster on virtual machines that run Red Hat Core OS. The Helper node provide services for the OpenShift Cluster as documented in the OpenShift 4 documentation: Helper Git Repository and Helper Blog. IBM implements two Helper virtual machines and uses a floating IP address to provide high availability for these services.
Bootstrap
There is one Bootstrap node that is used to install the OpenShift Container Platform control plane on the master nodes. It is only used during the bootstrapping of OpenShift Container Platform. Eventually, this virtual machine is destroyed and resources are released.
Master
There are three master nodes deployed on virtual machines that run on Red Hat Core OS. OpenShift 4 requires three master nodes, ensuring high availability and quorum of essential Kubernetes services like etcd.
Worker
By default, there are two worker nodes deployed on virtual machines that run on Red Hat Core OS. This ensures high availability of containers that run on these worker nodes. Depending on the needs for your OpenShift cluster, you could opt for a higher number of worker nodes or worker nodes with more CPU and memory. Starting from IBM Cloud Pak System V2.3.3.0, it is possible to add additional worker nodes to your OpenShift cluster after deployment (horizontal scaling).
As shown in the following table, by default, single OCP cluster requires 28 virtual CPUs, 112 GB of RAM, and 1202 GB of storage. Depending on the number and sizing of the worker nodes, the amount of resources required could be higher.
VM
|
Number
|
OS
|
virtual CPUs
|
RAM (GB)
|
storage (GB)
|
Primary Helper
|
1
|
RHEL 7.7
|
4
|
16
|
470
|
Secondary Helper
|
1
|
RHEL 7.7
|
4
|
16
|
12
|
Bootstrap
|
1
|
RH Core OS
|
4
|
16
|
120
|
Master
|
3
|
RH Core OS
|
4
|
16
|
120
|
Worker
|
2
|
RH Core OS
|
2
|
8
|
120
|
Total
|
8
|
–
|
28
|
112
|
1202
|
- Deployment of Red Hat OpenShift 4 cluster
With all the previous steps completed, you are now ready to deploy your first Red Hat OpenShift 4.6 cluster!
Go to Cloud Pak System user interface (https://<ICPS_system_IP>/cps/) and log in. The Getting started page opens up. Click Provision accelerators as shown in Figure 16.
Figure 16
The Provision accelerators page opens up. Search for OpenShift Container Platform and click Ready to deploy as shown in Figure 17.
Figure 17
User can deploy OpenShift Container Platform with default or customized configuration
You should see a page as shown in Figure 18. By default, OpenShift Container Platform version is set to 4.6.
Figure 18
Select Default or Customized option based on your requirement.
Default deployment
Default deployment option deploys an OpenShift Container Platform cluster with already configured values with minimum hardware OpenShift Container Platform cluster. To deploy default cluster, select Default as shown in Figure 19.
Figure 19
Customized deployment
Customized deployment enables you to configure the cluster as per business use case. It provides a way to change hardware specification for the OpenShift Container Platform cluster to be deployed. To deploy Customized cluster select Customized as shown in Figure 20 and continue.
Figure 20
If you want the customize deployment steps, refer to section Exploring Red Hat OpenShift 4 cluster accelerator section of this article. If you want to do a default deployment, continue with this procedure.
Click Continue to open the Configure deployment page as shown in Figure 21.
First section on this page shows environment profile, cloud group, and IP group for deployment. Select appropriate values based on where the RHUS and Cloud Pak System registry shared services are running.
Figure 21
In Helper node credentials section, specify password for root and virtuser as shown in Figure 22.
Optionally, in the Optional section as shown in Figure 23, specify a SSH key to connect to helper virtual machines via SSH post deployment. If not specified, then a default key is generated and used.
Figure 23
Click Review and deploy.
Review cluster topology and other values.
Click deploy as shown in Figure 24 to deploy cluster.
Within seconds you should see a message indicating that the deployment has started as shown in Figure 25.
Figure 25
In the message box, click Manage accelerator instances. You will be redirected to Manage accelerator instances page. As shown in figure 26, deployment starts with the cluster status in launching state.
Figure 26
It takes approximately 50 minutes to deploy OpenShift Container Platform cluster. After it is deployed, you should see an instance as shown in Figure 27.
Figure 27
Review history section of the instance for post deployment actions as highlighted in Figure 28.
Figure 28
Post deployment actions
Before you can use the OpenShift 4 cluster, a few more steps are required as documented in step 6 of Getting started with OpenShift Container Platform 4.x pattern.
Review history section for next steps as indicated in Figure 27.
- Retrieve the password for kubeadmin
The kubeadmin password gets generated during the installation of OpenShift 4. Retrieve kubeadmin password by providing root password for Helper VM as shown in Figure 29.
Figure 29
- Configure your DNS server
Set up the following two DNS wildcard entries for the floating IP address and fully-qualified domain name of your OpenShift 4 Virtual System Instance. This is required to access the OpenShift web-console, applications, and APIs.
*.<fqdn> IN A <ip>
*.apps.<fqdn> IN A <ip>
In the case of our OpenShift 4 cluster here, the floating IP address is ocp_cluster_ip with corresponding fully-qualified domain name cps-rack-79-vm-12.xxx.xxx.xxx.xxx. So you need to configure the following DNS wildcard entries:
*.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
*.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
If you are unable to easily make changes to your DNS server, you can add the following entries to your local /etc/hosts file (or equivalent on Windows) for testing purposes. This will allow you to log in to the OpenShift console, but note that you would need additional entries for any applications you would deploy later.
ocp_cluster_ip console-openshift-console.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx oauth-openshift.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx
You can find more information about OpenShift external DNS requirements here. The DNS records listed as “This record must be resolvable by both clients external to the cluster …” are required. DNS is also provided on the Helper Nodes to cover the resolution inside the cluster.
If you are able to configure DNS records up front, then the cluster console link will be accessible immediately. Configuring DNS ahead of time is the recommended approach for deploying OpenShift Container Platform clusters on Cloud Pak System. You would need to create the following records in your DNS server for each IP in the IP group you are using to deploy (so that any IP that is selected from the IP group to be the floating IP for the cluster will already have wildcard entries associated with it in DNS):
*.sub.domain IN A <ip>
*.mycluster.sub.domain IN A <ip>
- Access your Red Hat OpenShift cluster
You can now access your OpenShift 4 cluster using the OpenShift console link as shown in Figure 30.
Figure 30
Log in with the username kubeadmin and the password that you retrieved earlier as shown in Figure 31.
Figure 31
After you log in, you will see the console as shown in Figure 32.
Figure 32
Navigate to Compute > Nodes. If you see three master nodes and two worker nodes, then it confirms that the OpenShift 4 cluster topology was deployed as expected.
- Verify your OpenShift cluster
Inspect your cluster from IBM Cloud Pak Console -> Manage accelerator instances
In the IBM Cloud Pak Console -> Manage accelerator instances page, click Nodes for the deployed instance as shown in Figure 33, you can verify cluster topology and VMs deployed as part of instance.
Figure 33
In the IBM Cloud Pak Console -> Manage accelerator instances, click Middleware for the deployed instance as shown in Figure 34. You can verify the cluster topology and deployed VMs as a part of the instance.
Figure 34
NOTE: Do not stop or restart OpenShift Container Platform cluster or any nodes until after 24 hours of deployment. Failing to do so may render your cluster to a broken state that cannot be recovered. Refer this page for more details.
Register your Red Hat OpenShift cluster with Red Hat
Finally, do not forget to register your OpenShift cluster with Red Hat. This manual step is required if your OpenShift cluster does not have internet access to reach Red Hat. You can follow step 4 here to register your cluster on the “Cluster registration” page.
Next steps
Now you are ready for day 2 operations and workload deployment on your OpenShift Container Platform cluster. See this article for day 2 OpenShift Container Platform cluster operations in IBM Cloud Pak System.
Conclusion
IBM Cloud Pak System 2.3.3.3 enables clients to quickly roll out one or more Red Hat OpenShift 4.6 clusters, which greatly simplifies the process, ensures consistency, and avoids human error. This is also used as the foundation for the deployment of IBM Cloud Paks on the IBM Cloud Pak System platform.
I would like to thank Sanjeev Pradhan for his help in writing this article.