Overview
IBM Cloud Pak System accelerates your implementation of on-premises Kubernetes platforms. It comes with support for automated deployment and configuration of Red Hat OpenShift Container Platform (OCP). This makes it the perfect platform for on-premises deployment of IBM Cloud Paks and Red Hat OpenShift clusters.
This tutorial walks you through steps for deployment of Red Hat OpenShift Container Platform 4.4 on IBM Cloud Pak System V2.3.3.0.
Introduction
This tutorial focuses on the deployment of Red Hat OpenShift Container Platform 4.4. For details on version 4.3, refer to the IBM Developer article Deploying Red Hat OpenShift 4.3 on IBM Cloud Pak System.
For Red Hat OpenShift, it is important to know that there are several different offerings available:
A fully managed public cloud offering for quickly deploying applications.
- OpenShift Hosted Services
OpenShift clusters hosted on IBM Cloud, Amazon Web Services (AWS), and Azure.
- OpenShift Container Platform (OCP)
An enterprise OpenShift cluster deployed on your own on-premises infrastructure (OpenShift Container Platform was previously called OpenShift Enterprise, but the name was changed with the release of version 3.3.).
A more detailed comparison of these offerings can be found on the OpenShift website. As IBM Cloud Pak System is an on-premises appliance, it only provides support for the OpenShift Container Platform offering. In this tutorial, you will learn how to deploy OpenShift Container Platform on IBM Cloud Pak System. Steps are written assuming that the IBM Cloud Pak System is at 2.3.3.0 firmware, and does not have direct access to the internet.
Prerequisites
Before you deploy your first OpenShift 4.4 cluster on IBM Cloud Pak System, few prerequisites need to be in place. For a good starting point on these prerequisites, see IBM Knowledge Center:
- IBM Cloud Pak System 2.3.3.0 Intel based IBM Cloud Pak System models W2500, W3500 and W3550 are supported. There is currently no support for the Power based IBM Cloud Pak System model W3700.
- IBM OS image for Red Hat Linux Systems (RHEL 7.7 X64) Version 7.7 Scenarios using a custom OS image are also supported, as long as it is Red Hat Enterprise Linux (RHEL) 7.7 or higher.
- IBM OS image for Red Hat Enterprise Linux CoreOS Version 4.4.0 This is included with accelerator bundle for OpenShift Container Platform 4.4.0.0.
- Docker Accelerator 1.0.13.0 Docker registry is required for accessing OpenShift Container Platform images. Obtain Docker registry accelerator 1.0.13.0 from IBM FixCentral.
- OpenShift Container Platform Accelerator 4.4.0.0 OpenShift Container Platform accelerator is used for deploying OpenShift Container Platform cluster in IBM Cloud Pak System. Get OpenShift Container Platform accelerator 4.4.0.0 from IBM FixCentral.
- Red Hat Satellite Server 6 shared service deployed The shared service should be connected to an existing Red Hat Satellite Server (RHSS), or to RHSS deployed on IBM Cloud Pak System with access to repositories rhel-7-server-rpms & rhel-7-server-extras-rpms.
Note that IBM Cloud Pak System comes with Red Hat subscriptions for RHEL and RHSS.
- Active subscription with Red Hat for the OpenShift Container Platform Unlike the Red Hat subscription for RHEL and RHSS, the OpenShift Container Platform (OCP) subscription is not included with IBM Cloud Pak System. Please refer to OpenShift Container Platform accelerator V4.3.1.0 recipe for detailed steps.
- Sufficient compute, memory and storage resources on IBM Cloud Pak System A single OpenShift Container Platform cluster requires at least 28 virtual CPUs, 112 GB of RAM, and 1202 GB of storage.
For detailed prerequisite steps to configure Red Hat Satellite Server and Private Docker Registry on IBM Cloud Pak System, refer to OpenShift Container Platform accelerator V4.3.1.0 recipe.
Step-by-step
- Load and verify required artefacts on IBM Cloud Pak System V2.3.3.0
By default, IBM Cloud Pak System 2.3.3.0 includes most of the required content pre-loaded for you.
For the sake of completeness, see the following complete list of everything that you need:
- Verifying and importing BYOL binaries
The “IBM Cloud Pak System accelerator bundle for Red Hat Openshift V4.4.0” are not installed by default. To confirm, log in to IBM Cloud Pak System and go to System > Storehouse Browser. If you do not see an entry for /admin/files/RedHatOpenShift as shown in Figure 1, then it means that the binaries are not installed yet.
Figure 1
Process for loading the binaries has been documented here in the IBM Cloud Pak System Knowledge Center.
/****************************************************
Cloud Pak Accelerator Bundle Import utility
***************************************************
Artifacts from accelerator Bundle will be imported to Cloud Pak System
Cloud Pak System hostname or IP address: 9.XX.XX.XX
Cloud Pak System username: admin
Password for admin:
- Testing connectivity to Cloud Pak System
Upload Cloud Pak binaries to Storehouse
-----------------------------------------------
- Uploading binaries for: Redhat Openshift 4.4.6 on IBM Cloud Pak System
- Pre-upload Verification
1) openshift-install-linux-4.3.1.tar.gz: verified successfully
2) ocp4.4.6-x86_64.tgz: verified successfully
3) openshift-install-linux-4.4.6.tar.gz: verified successfully
4) openshift-client-linux-4.4.6.tar.gz: verified successfully
5) ocp4.3.1-x86_64.tgz: verified successfully
6) ocp4.3.1-x86_64-extra.tar: verified successfully
7) jq-linux64: verified successfully
8) openshift-client-linux-4.3.1.tar.gz: verified successfully
- All binaries verified successfully.
- openshift-install-linux-4.3.1.tar.gz is already on the server
- ocp4.4.6-x86_64.tgz is already on the server
- openshift-install-linux-4.4.6.tar.gz is already on the server
- openshift-client-linux-4.4.6.tar.gz is already on the server
- ocp4.3.1-x86_64.tgz is already on the server
- ocp4.3.1-x86_64-extra.tar is already on the server
- jq-linux64 is already on the server
- openshift-client-linux-4.3.1.tar.gz is already on the server
- Verifying file
Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
- Verifying file
Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
- Downloading CLI from Cloud Pak System
- Starting Cloud Pak System CLI
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/jna8015534376793836385.tmp which might have disabled stack guard. The VM will try to fix the stack guard now.
It’s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack’.
Import and Clone Virtual Images
----------------------------------------
- Checking for Virtual Image
Name: IBM OS Image for Red Hat Linux Systems
Version: 3.1.0.0
- Virtual Image is available
- Working with image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
Version: 4.3.0
- Image exists. Skipping importing the image
- Cloned Image exists. Skipping cloning of image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
- Cloned Image exists. Skipping cloning of image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 250G
- Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
- Working with image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
Version: 4.4.3
- Image exists. Skipping importing the image
- Cloned Image exists. Skipping cloning of image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
- Cloned Image exists. Skipping cloning of image
Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 250G
- Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
Import Ptypes
------------------------
- Import ptype for: Redhat Openshift 4.4.6 on IBM Cloud Pak System
- Working with ptype
Name: openshift
Version: 4.4.0.0
- Ptype exists. Skipping the Ptype import*/
Confirm that the following files are now visible from the IBM Cloud Pak System. Go to System > Storehouse browser and confirm that you see what is shown in Figure 2.
Figure 2
New and improved Cloud Pak accelerator bundle import utility is available for OpenShift Container Platform V4.4.0.0 accelerator on IBM Cloud Pak System. It loads images to storehouse, clones the Red Hat Enterprise Linux CoreOS OVA to three variants with different disk size of 16 GB, 120 GB, and 250 GB, and makes them available in IBM Cloud Pak System Virtual Images catalog as shown in Figure 3. In addition, it also imports the accelerator into IBM Cloud Pak System.
Figure 3
- Verifying and importing Pattern Types
The new and improved Cloud Pak accelerator bundle import utility (described in previous step) for IBM Cloud Pak System loads pattern type to IBM Cloud Pak System too. After it is loaded, you should see the OpenShift 4.4.0.0 accelerator in the catalog with status available as shown in Figure 4.
Figure 4
Exploring Red Hat OpenShift 4 cluster accelerator
Navigate to Provision environments from IBM Cloud Pak System user interface as shown in Figure 5.
Figure 5
Search for OpenShift Container Platform and click Ready to deploy as shown in Figure 6.
Figure 6
This action opens a page to select Default or Customized cluster.
Select Customized as shown in Figure 7 and click Continue. The Configure deployment page opens.
Figure 7
The first section in this page includes IBM Cloud Pak System deployment options as shown in Figure 8. For example, environment profile, cloud group, and so on. Select the appropriate values such that all prerequisites are available in the chosen cloud group.
Figure 8
The second section in this page contains OS users root and virtuser credentials for helper virtual machines as shown in Figure 9.
Figure 9
The last section in this page includes an optional parameter to generate SSH key, which is used in deployment as shown in Figure 10. If you generate this key, you can use it to connect to helper and OpenShift Container Platform virtual machines via SSH.
Figure 10
Click Configure nodes to open the Configure nodes tab page.
This page shows type of VMs, such as helpers, control planes and worker nodes, which are included in the accelerator and hardware resource configuration for each virtual machine as shown in Figure 11 and Figure 12. You can change the resource configuration of all nodes in a cluster to be deployed. Use slider, as shown in Figure 11, to set the desired value for each resource type of virtual machines.
Figure 11
Note: You cannot change number of masters to be deployed as Red Hat recommends deploying three masters for OpenShift Container Platform 4 clusters.
Figure 12
Click OpenShift options to open the OpenShift options tab page.
This page shows deployment parameters for OpenShift Container Platform accelerator as shown in Figures 13 and 14. Detailed description of parameters can be found here.
It is the Red Hat OpenShift Container Platform version to deploy.
- OpenShift cluster domain name
It is the domain name for Red Hat OpenShift Container Platform cluster. If not specified, then a default cluster domain name is generated.
It is the Red Hat OpenShift Container Platform cluster name. If not specified, then a default cluster name is generated.
- OpenShift image registry name
It is the fully qualified name for the Red Hat OpenShift Container Platform image registry. In case there is no Cloud Pak System Registry available in cloud group or you wish to use a different docker registry for accessing OpenShift Container Platform images, specify registry hostname:port here.
- OpenShift image registry username
It is the username needed to access the Red Hat OpenShift Container Platform secure image registry.
- OpenShift image registry password
It is the user password that is needed to access the Red Hat OpenShift Container Platform secure image registry.
If access to external OpenShift registry is available and you wish to use it for OpenShift Container Platform images, paste contents of the downloaded OpenShift install pull-secret.json file here.
Figure 13
- Alternate NFS server name or IP address (for the OpenShift image registry)
It is the alternate NFS server name or IP address of the OpenShift image registry. By default, Primary helper virtual machine is used to configure storage for OpenShift image registry. If this parameter value is specified, then this external NFS server is used for OpenShift image registry instead. Please see this page about configuring NFS server and path.
- Alternate NFS server path (for the OpenShift image registry)
It is the alternate NFS server path of the OpenShift image registry. It is used along with the Alternate NFS server name or IP address parameter to configure storage for OpenShift image registry.
- Alternate NFS server name or IP address (for the application persistent storage)
It is the alternate NFS server name or IP address of the Application Persistent Storage. By default Primary helper is used to configure persistent storage for workloads to be deployed on OpenShift Container Platform cluster. When value is specified here, it is used for persistent storage for workloads instead.
- Alternate NFS server path (for the application persistent storage)
It is the alternate NFS server path of the Application Persistent Storage. It is used along with Alternate NFS server name or IP address parameter to configure persistent storage for workloads.
Figure 14
Click Review and deploy to open the Ready to deploy tab page. This page shows a summary of topology and resource configuration as shown in Figure 15.
Figure 15
Following is a brief overview of various nodes included in cluster.
- PrimaryHelper and SecondaryHelper
There are two Helper nodes that run on RHEL 7.7. It supports the deployment of OpenShift cluster on virtual machines that run Red Hat Core OS. The Helper node provide services for the OpenShift Cluster as documented in the OpenShift 4 documentation: Helper Git Repository and Helper Blog. IBM implements two Helper virtual machines and uses a floating IP address to provide high availability for these services.
There is one Bootstrap node that is used to install the OpenShift Container Platform control plane on the Master nodes. It is only used during the bootstrapping of OpenShift Container Platform. Eventually, this virtual machine is destroyed and resources are released.
There are three Master nodes deployed on virtual machines that run on Red Hat Core OS. OpenShift 4 requires three Master nodes, ensuring high availability and quorum of essential Kubernetes services like etcd.
By default, there are two Worker nodes deployed on virtual machines that run on Red Hat Core OS. This ensures high availability of containers that run on these Worker nodes. Depending on the needs for your OpenShift cluster, you could opt for a higher number of Worker nodes or Worker nodes with more CPU and memory. Starting from IBM Cloud Pak System V2.3.3.0, it is possible to add additional Worker nodes to your OpenShift cluster after deployment (horizontal scaling).
As shown in the following table, by default, single OCP cluster requires 28 virtual CPUs, 112 GB of RAM and 1202 GB of storage. Depending on the number and sizing of the worker nodes, the amount of resources required could be higher.
|
VM
|
Number
|
OS
|
virtual CPUs
|
RAM (GB)
|
storage (GB)
|
|
Primary Helper
|
1
|
RHEL 7.7
|
4
|
16
|
470
|
|
Secondary Helper
|
1
|
RHEL 7.7
|
4
|
16
|
12
|
|
Bootstrap
|
1
|
RH Core OS
|
4
|
16
|
120
|
|
Master
|
3
|
RH Core OS
|
4
|
16
|
120
|
|
Worker
|
2
|
RH Core OS
|
2
|
8
|
120
|
|
Total
|
8
|
–
|
28
|
112
|
1202
|
Deployment of Red Hat OpenShift 4 cluster
With all the previous steps completed, you are now ready to deploy your first Red Hat OpenShift 4.4 cluster!
Go to Cloud Pak System user interface (https://<ICPS_system_IP>/cps/) and log in. The Getting started page opens as shown in Figure 16.
Figure 16
Click Provision environment as shown in Figure 17.
Figure 17
The Provision environment page opens as shown in Figure 18.
Figure 18
Search for OpenShift Container Platform and click Ready to deploy as shown in Figure 19.
Figure 19
User can deploy OpenShift Container Platform with default or customized configuration
You should see a page as shown in Figure 20. By default, OpenShift Container Platform version is set to 4.4.
Figure 20
Select Default or Customized option based on your requirement.
Default deployment option deploys an OpenShift Container Platform cluster with already configured values with minimum hardware OpenShift Container Platform cluster. To deploy default cluster, select Default as shown in Figure 21.
Figure 21
Customized deployment enables you to configure the cluster as per business use case. It provides a way to change hardware specification for the OpenShift Container Platform cluster to be deployed. To deploy Customized cluster select Customized as shown in Figure 22 and continue.
Figure 22
If you want the customize deployment steps, refer to section Exploring Red Hat OpenShift 4 cluster accelerator section of this article. If you want to do a default deployment, continue with this procedure.
Click Continue to open the Configure deployment page as shown in Figure 23.
First section on this page shows environment profile, cloud group, and IP group for deployment. Select appropriate values based on where the RHUS and Cloud Pak System registry shared services are running.
Figure 23
In Helper node credentials section, specify password for root and virtuser as shown in Figure 24.
Figure 24
Optionally, in the Optional section as shown in Figure 25, specify a SSH key to connect to helper virtual machines via SSH post deployment. If not specified, then a default key is generated and used.
Figure 25
Click Review and deploy as shown in Figure 26.
Figure 26
Review cluster topology and other values.
Click deploy as shown in Figure 27 to deploy cluster.
Figure 27
Within seconds you should see a message indicating that the deployment has started as shown in Figure 28. 10. In the message box, click Manage environments. You will be redirected to Manage environments page.
Figure 28
As shown in figure 29, deployment starts with the cluster status in launching state.
Figure 29
It takes approximately 50 minutes to deploy OpenShift Container Platform cluster. After it is deployed, you should see an instance as shown in Figure 30.
Figure 30
Review history section of the instance for post deployment actions as highlighted in Figure 31.
Figure 31
Post deployment actions
Before you can use the OpenShift 4 cluster, a few more steps are required as documented in step 6 of Getting started with OpenShift Container Platform 4.x pattern.
Review history section for next steps as indicated in Figure 30.
- Retrieve the password for kubeadmin
The kubeadmin password gets generated during the installation of OpenShift 4. Retrieve kubeadmin password by providing root password for Helper VM as shown in Figure 32.
Figure 32
- Configure your DNS server
Set up the following two DNS wildcard entries for the floating IP address and fully-qualified domain name of your OpenShift 4 Virtual System Instance. This is required to access the OpenShift web-console, applications, and APIs.
*.<fqdn> IN A <ip>
*.apps.<fqdn> IN A <ip>
In the case of our OpenShift 4 cluster here, the floating IP address is ocp_cluster_ip with corresponding fully-qualified domain name cps-rack-79-vm-12.rtp.raleigh.ibm.com. So you need to configure the following DNS wildcard entries:
*.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
*.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
If you are unable to easily make changes to your DNS server, you can add the following entries to your local /etc/hosts file (or equivalent on Windows) for testing purposes. This will allow you to log in to the OpenShift console, but note that you would need additional entries for any applications you would deploy later.
ocp_cluster_ip console-openshift-console.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx oauth-openshift.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx
You can find more information about OpenShift external DNS requirements here. The DNS records listed as “This record must be resolvable by both clients external to the cluster …” are required. DNS is also provided on the Helper Nodes to cover the resolution inside the cluster.
If you are able to configure DNS records up front, then the cluster console link will be accessible immediately. Configuring DNS ahead of time is the recommended approach for deploying OpenShift Container Platform clusters on Cloud Pak System. You would need to create the following records in your DNS server for each IP in the IP group you are using to deploy (so that any IP that is selected from the IP group to be the floating IP for the cluster will already have wildcard entries associated with it in DNS):
*.sub.domain IN A <ip>
*.mycluster.sub.domain IN A <ip>
- Access your OpenShift cluster
You can now access your OpenShift 4 cluster using the OpenShift console link as shown in Figure 33.
Figure 33
Log in with the username kubeadmin and the password that you retrieved earlier as shown in Figure 34.
Figure 34
After you log in, you will see the console as shown in Figure 35.
Figure 35
Navigate to Compute > Nodes. If you see three Master nodes and two Worker nodes, then it confirms that the OpenShift 4 cluster topology was deployed as expected.
Verify your OpenShift cluster
Inspect your cluster from IBM Cloud Pak Console -> Manage environments
In the IBM Cloud Pak Console -> Manage environments page, click Nodes for the deployed instance as shown in Figure 36, you can verify cluster topology and VMs deployed as part of instance.
Figure 36
In the IBM Cloud Pak Console -> Manage environments, click Middleware for the deployed instance as shown in Figure 37. You can verify the cluster topology and deployed VMs as a part of the instance.
Figure 37
NOTE: Do not stop or restart OpenShift Container Platform cluster or any nodes until after 24 hours of deployment. Failing to do so may render your cluster to a broken state that cannot be recovered. Refer this page for more details.
Register your OpenShift cluster with Red Hat
Finally, do not forget to register your OpenShift cluster with Red Hat. This manual step is required if your OpenShift cluster does not have internet access to reach Red Hat. You can follow step 4 here to register your cluster on the “Cluster registration” page.
Next steps
Now you are ready for day 2 operations and workload deployment on your OpenShift Container Platform cluster. See this article for day 2 OpenShift Container Platform cluster operations in IBM Cloud Pak System.
Conclusion
IBM Cloud Pak System 2.3.3.0 enables clients to quickly roll out one or more Red Hat OpenShift 4.4 clusters, which greatly simplifies the process, ensures consistency, and avoids human error. This is also used as the foundation for the deployment of IBM Cloud Paks on the IBM Cloud Pak System platform.
I would like to thank fellow IBMers Hina Sharma, Shreya Kunar and Shyamala Rajagopalan for their help in creating this tutorial.