Introduction
Red Hat OpenShift Container Platform (RHOCP) is Red Hat’s enterprise Kubernetes distribution. It provides developers, IT organizations, and business leaders with a hybrid cloud application platform for developing and deploying new containerized applications.
This blog article is part of a series on how to set up a cluster with Red Hat OpenShift Virtualization (RHOCPV). This article describes how to set up an RHOCP cluster in LPARs on IBM Z and LinuxONE which can be used as a base for installing RHOCPV. The setup of RHOCPV and how to use it to deploy virtual machines is described in part 2 of this blog post series.
RHOCP on LPAR nodes
IBM Z and LinuxONE can host RHOCP clusters with different types of hypervisors that host the nodes. If you intend to run RHOCPV, the nodes that host virtual machines need to be hosted directly in logical partitions (LPARs) without an additional hypervisor (z/VM or KVM) running in the LPAR. RHOCP also comes in different control plane topologies:
- Full high-availability deployment with three dedicated control plane nodes
- Three-Node cluster with three converged control plane nodes that are also compute nodes
- Single-node OpenShift with one control plane node that is also a compute node
While all of these control planes topologies can be scaled with additional dedicated compute nodes, the choice of having dedicated control planes or converged nodes and the number of nodes has impact on the availability of the control plane in case of outages or peak load scenarios. You can find more information on the control plane topology on IBM Z and IBM LinuxONE in the IBM Red Hat OpenShift for IBM Z docs. With both these architectural decisions in mind, we will walk through the setup of a full high-availability cluster with three dedicated control plane nodes and two compute nodes that can host both containers as well as virtual machines using RHOCPV. All of the nodes will be ran directly in LPARs on IBM Z.
With both these architectural decisions in mind, we will walk through the setup of a full high-availability cluster with three dedicated control plane nodes and two compute nodes that can host both containers as well as virtual machines using RHOCPV. All of the nodes will be ran directly in LPARs on IBM Z.
Prerequisites
You can run the setup described in this article on the following IBM® hardware:
- IBM® z17 (all models), IBM® z16 (all models), IBM® z15 (all models), IBM® z14 (all models)
- IBM® LinuxONE 5 (all models), IBM® LinuxONE 4 (all models), IBM® LinuxONE III (all models), IBM® LinuxONE Emperor II, IBM® LinuxONE Rockhopper II
The system requirements are:
- The equivalent of six Integrated Facilities for Linux (IFL), which are SMT2 enabled. You can use either shared or dedicated IFLs.
- Five logical partitions (LPARs)
- One temporary machine for bootstrapping the cluster installation. You can use one of the LPARs used for compute nodes for this purpose.
- Direct attached OSA or RoCE network adapters.
Storage / Main Memory requirements:
- 16 GB for OpenShift Container Platform control plane machines
- 8 GB for OpenShift Container Platform compute machines
- 16 GB for the temporary OpenShift Container Platform bootstrap machine.
In this description, we will describe how to address FCP disks attached to the LPARs. If you need a different type of storage setup, please refer to the docs on details how to configure OpenShift to use other types of storage.
Environment Setup
Hosting an OpenShift cluster inside your environment requires some auxiliary services: name resolution via DNS, an external load balancer and file servers for hosting boot artifacts. You can find details on the required environment in the Red Hat OpenShift documentation. While it varies depending on the existing environment and available software, we will describe how to set up essential services in a machine we will call bastionfor the purpose of this blog post. If you are using different DNS services, load balancers and file servers, you can partially skip this description and follow the setup description in the linked docs.
Bastion Setup
Set up a machine that is running Red Hat Enterprise Linux (RHEL). You can also host bastion inside an LPAR on IBM Z. On bastion, we will host name resolution and reverse name resolution via BIND (named), load balancing via haproxy, web server via Apache httpd and FTP server via vsftpd.
Name Resolution via BIND Setup
We will be using BIND (Berkley Internet Domain Server) for everything pertaining to name resolution. Particularly, we will be using named as a DNS server.
# dnf install bind
Create a bind configuration in /etc/named.conf. Add the following to the end of the file:
zone "ocp.example.com" {
type master;
file "ocp.example.com.zone";
allow-query { any; };
allow-transfer { none; };
allow-update { none; };
};
zone "0.168.192.in-addr.arpa" {
type master;
file "0.168.182.in-addr.arpa.zone";
allow-query { any; };
allow-transfer { none; };
allow-update { none; };
};
If you want to specify a DNS server to fall back to for requests not inside ocp.example.com, you can specify a section inside the options { ... } of the configuration. Depending on the requirement, you might need to disable DNSSEC or do additional configuration to enable it for your system. In the example, it is disabled:
options {
forward only;
forwarders {
/* ip of the forwarder */;
};
dnssec-enable no;
dnssec-validation no;
};
To make it possible for all LPARs to send queries to the DNS server, please configure the following options in the options { ... } section of the configuration:
options {
listen-on port 53 { 127.0.0.1; 192.168.0.100; }; Add the IP of your bastion here, so the DNS also listens to queries from ouside.
...
allow-query { Add the IPs of your guests, or any.
localhost;
192.168.0.100;
192.168.0.110;
192.168.0.111;
192.168.0.112;
192.168.0.120;
192.168.0.121;
192.168.0.122;
};
};
Create a subnet definition in /var/named/ocp.example.com.zone:
$TTL 900
@ IN SOA bastion.ocp.example.com. hostmaster.ocp.example.com. (
2019062002 1D 1H 1W 3H
)
IN NS bastion.ocp.example.com.
bastion IN A 192.168.0.100
api IN A 192.168.0.100
api-int IN A 192.168.0.100
apps IN A 192.168.0.100
*.apps IN A 192.168.0.100
control0 IN A 192.168.0.110
control1 IN A 192.168.0.111
control2 IN A 192.168.0.112
compute0 IN A 192.168.0.120
compute1 IN A 192.168.0.121
bootstrap IN A 192.168.0.122
Create a DNS subnet zone in /var/named/0.168.192.in-addr.arpa.zone:
$TTL 900
@ IN SOA bastion.ocp.example.com hostmaster.ocp.example.com. (
2019062001 1D 1H 1W 3H
)
IN NS bastion.ocp.example.com.
110 IN PTR control0.ocp.example.com.
111 IN PTR control1.ocp.example.com.
112 IN PTR control2.ocp.example.com.
120 IN PTR compute0.ocp.example.com.
121 IN PTR compute1.ocp.example.com.
122 IN PTR bootstrap.ocp.example.com.
Make sure that your resolv.conf (/etc/resolv.conf) points to localhost as the DNS server and contains the following line:
nameserver 127.0.0.1
Ensure that the firewall allows DNS requests and start and enable the named service. If you have already started the service, you can restart it or reload the configuration.
# firewall-cmd --add-service=dns --zone=public --permanent
# firewall-cmd --reload
# systemctl start named && systemctl enable named
You can use the tool dig from the package bind-utils to verify that name resolution works:
# dig @<ip of your machine> control0.ocp.example.com
// output should show a DNS record that contains the correct IP address
Load Balancer Setup
We will be using HAProxy as the external load balancer for the cluster.
First, we install the haproxy package:
# dnf install haproxy
For demonstration purposes, we will allow haproxy to bind to any port. In an production environment, this should be scoped to specific ports.
# setsebool -P haproxy_connect_any=1
Add the following configuration to your /etc/haproxy/haproxy.cfg, remove othre listen blocks:
listen api-server-6443
bind *:6443
mode tcp
option httpchk GET /readyz HTTP/1.0
option log-health-checks
balance roundrobin
server bootstrap bootstrap.ocp.example.com:6443 verify none check check-ssl inter 10s fall 2 rise 3 backup
server control0 control0.ocp.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
server control1 control1.ocp.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
server control2 control2.ocp.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
listen machine-config-server-22623
bind *:22623
mode tcp
server bootstrap bootstrap.ocp.example.com:22623 check inter 1s backup
server control0 control0.ocp.example.com:22623 check inter 1s
server control1 control1.ocp.example.com:22623 check inter 1s
server control2 control2.ocp.example.com:22623 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server compute0 compute0.ocp.example.com:443 check inter 1s
server compute1 compute1.ocp.example.com:443 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server compute0 compute0.ocp.example.com:80 check inter 1s
server compute1 compute1.ocp.example.com:80 check inter 1s
Configure the firewall to allow access to these services:
# firewall-cmd --add-service=http --zone=public --permanent
# firewall-cmd --add-service=https --zone=public --permanent
# firewall-cmd --add-port=6443/tcp --zone=public --permanent
# firewall-cmd --add-port=22623/tcp --zone=public --permanent
# firewall-cmd --reload
Enable the haproxy service:
# systemctl start haproxy && systemctl enable haproxy
Web Server Setup
As web server, we will be using the Apache HTTP server as provided by the httpd package. To install it, we install the respective package:
# dnf install httpd
We have to change the port it is listening to, since port 80 is already bound by HAProxy for cluster ingress. To bind to port 8080 instead, change the line Listen 80 in /etc/httpd/conf/httpd.conf to Listen 8080.
Add the required rules to the firewall:
# firewall-cmd --add-port=8080/tcp --zone=public --permanent
# firewall-cmd --reload
Start the service:
# systemctl start httpd && systemctl enable httpd
FTP Server Setup
For this guide we will use Very Secure FTP Daemon (vsftpd). We will do a setup similar to what is described in the Red Hat RHEL documentation for preparing network-based repositories for RHEL installations. It is also possible to set up SSH access and serve the files for the LPAR boot via SFTP instead.
Install the vsftpd package:
# dnf install vsftpd
In the config file (/etc/vsftpd/vsftpd.conf), change the following settings (or create them if they do not exist):
anonymous_enable=YES
write_enable=NO
pasv_min_port=10021
pasv_max_port=10031
Create the appropriate firewall rules to allow access to the FTP server and start the vsftpd service:
# firewall-cmd --add-port 21/tcp --permanent
# firewall-cmd --add-port 10021-10031/tcp --permanent
# firewall-cmd --add-service ftp --permanent
# firewall-cmd --reload
# systemctl start vsftpd && systemctl enable vsftpd
This concludes the setup of the bastion.
Red Hat OpenShift Setup
Obtaining the Installation Artifacts
To run the setup, you need to obtain the OpenShift installer from the Red Hat Hybrid Cloud Console. Navigate to the page for Red Hat OpenShift Container Platform and choose the “Datacenter” tab, then IBM Z (s390x). (Direct link to "Create an OpenShift Cluster: IBM Z (s390x)" in the Red Hat Hybrid Cloud Console).
On the following page, you will find all of the necessary artifacts you require to go through the installation.
Download the installer (1), pull secret (2) and CLI (3) to the machine you intend to run the installer CLI on. Extract the installer and CLI and add all of the files to your PATH. For our article, we will run all of these from the bastion node, so we will use Linux s390x for both the installer and CLI. For the LPAR installation, we download the live ISO from the OpenShift mirror repository. Choose the rhcos-live.s390x.iso and download it to your bastion machine:
# mkdir -p /root/artifacts && cd /root/artifacts
# curl -LO https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/latest/rhcos-live-iso.s390x.iso
...
Customizing LPAR Boot Artifacts
We will boot each LPAR node with custom kernel parameter file to specify network configuration and storage configuration. To this end, we first mount the live iso file, since it contains all necessary artifacts for booting an LPAR.
# cd /root/artifacts
# mkdir rhcos-iso
# mount -o loop,ro ./rhcos-live-iso.s390x.iso ./rhcos-iso
# cp -r rhcos-iso rhcos-artifacts
# umount ./rhcos-live-iso.s390x.iso && rm -rf ./rhcos-iso ./rhcos-live.s390x.iso
For simplicity, we create a customized folder for each of the LPARs. Repeat the following for each LPAR (bootstrap, control0, control1, control2, compute0, compute1):
1. Copy rhcos-artifacts
# cp -r rhcos-artifacts control0
# chmod -R u+w control0
2. Edit control0/images/genericdvd.prm as follows:
cio_ignore=all,!condev rd.neednet=1 console=ttysclp0 coreos.inst.install_dev=/dev/disk/by-id/scsi-<serial_number> coreos.live.rootfs_url=http://bastion.ocp.example.com:8080/images/rhcos-live-rootfs.s390x.img coreos.inst.ignition_url=http:// bastion.ocp.example.com:8080/ignition/master.ign ip=192.168.0.110::192.168.0.1:255.255.255.0:::none nameserver=192.168.0.100 rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000
Make sure to configure the SCSI-serial number and FCP device address correctly in the configuration. Use master.ign, bootstrap.ign and worker.ign for the different types of nodes and adjust the IP address as specified in the DNS configuration. If you have multiple network adapters configured and run into issues with the network not coming up correctly, please add the name of the main interface to use to the ip= parameter as follows: ip=192.168.0.110::192.168.0.1:255.255.255.0::ensXXX:none.
Generating Ignition Files
Each of the LPARs will be running Red Hat Enterprise Linux CoreOS (RHCOS) to make up the RHOCP cluster. The boot process of RHOCS is controlled via a so-called Ignition file which describes exactly the steps on how the operating system needs to be automatically set up and connected to the cluster.
The ignition files are generated using the openshift-install binary obtained from the Hybrid Cloud Console. The input for openshift-install is an install-config.yaml that described parameters of the OpenShift cluster to create. We are using a customized install-config.yaml based on the description in the OpenShift docs. You can find all installation configuration parameters for IBM Z and LinuxONE in the docs.
Create a file called install-config.yaml in a suitable working directory. We will use /root/ocp4-workdir
apiVersion: v1
baseDomain: example.com
compute:
- hyperthreading: Enabled
name: worker
replicas: 0
architecture: s390x
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
architecture: s390x
metadata:
name: ocp
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA...'
Note that the fields metadata.name and the baseDomain must make up the cluster domain we have set up for the nodes previously (in this blog article ocp.example.com).
Fill in the pull secret in the install-config.yaml, as well as a public SSH key for connecting to the cluster nodes. To generate the manifests, use openshift-install (if you have not added openshift-install to your PATH, first navigate to the folder of the binary and use ./openshift-install instead):
# openshift-install --dir=/root/ocp4-workdir create manifests
# openshift-install --dir=/root/ocp4-workdir create ignition-configs
Hosting All Artifacts on Bastion
For simplicity in this article, we will create duplicates of required boot artifacts. You can also deduplicate files and structure your .ins files to point to the files.
Move all of the created LPAR boot artifacts to the FTP root:
# mkdir -p /var/ftp/rhocp/
For each of the LPARs repeat the following while adapting the paths accordingly:
# mkdir -p /var/ftp/rhocp/bootstrap/images
# cp /root/artifacts/bootstrap/generic.ins /var/ftp/rhocp/bootstrap/
# cp /root/artifacts/bootstrap/images/genericdvd.prm /var/ftp/rhocp/bootstrap/images/
# cp /root/artifacts/bootstrap/images/initrd.addrsize /var/ftp/rhocp/bootstrap/images/
# cp /root/artifacts/bootstrap/images/pxeboot/{kernel,initrd}.img /var/ftp/rhocp/bootstrap/images/
Ensure that all the files are available in the correct structure for each LPAR:
# ls /var/ftp/rhocp/bootstrap/
generic.ins images
# ls /var/ftp/rhocp/bootstrap/images/
genericdvd.prm initrd.addrsize initrd.img kernel.img
Copy the rootfs.img from rhcos-artifacts and all of the ignition files to the correct directory to make it available via httpd:
# mkdir -p /var/www/html/images
# cd /var/www/html/images
# curl -LO https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/latest/rhcos-live-rootfs.s390x.img
# mkdir -p /var/www/html/ignition/
# cp /root/ocp4-workdir/*.ign /var/www/html/ignition/
# chmod go+r /var/www/html/ignition/*.ign
Booting the LPARs
You can boot the LPARs using the Hardware Management Console (HMC) of your IBM Z or IBM LinuxONE. Log into the HMC and select the LPAR you want to boot. Activate the LPAR before continuing. For the LPAR, select "Recovery" and "Load from Removable Media or Server".
Provide the data for allowing the HMC to connect to your FTP server with anonymous credentials.
Select the correct generic.ins file for booting the LPAR.
Finishing the Installation
After booting the LPARs, the cluster will go through the bootstrap process to set up the required resources and orchestrate the setup. You can monitor the bootstrap process using openshift-install as follows:
# openshift-install --dir /root/ocp4-workdir wait-for bootstrap-complete --log-level=info
The previous installation steps have set up the required authentication artefacts in /root/ocp4-workdir/auth/kubeconfig. You can copy this kubeconfig to the default path:
# mkdir -p ~/.kube/config
# cp /root/ocp4-workdir/auth/kubeconfig ~/.kube/config
Because we are installing without any machine API (platform none), you need to approve Certificate Signing Requests (CSRs) for the compute nodes of the cluster as they join the cluster. To do that, please watch incoming CSRs, e.g. as follows:
# watch -n5 oc get csr
You can approve the single CSRs for the nodes via:
# oc adm certificate approve <csr name>
To approve all currently pending CSRs, run the following command, as described in the OpenShift documentation:
# oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}} {{end}}{{end}}' | xargs oc adm certificate approve
The cluster is ready, when all cluster operators have successfully initialized and all nodes are ready. You can watch the cluster operators as follows:
# watch -n5 oc get clusteroperators
You can also use the openshift-install binary to wait for the cluster to be fully installed as follows:
# openshift-install --dir /root/ocp4-workdir wait-for install-complete
Conclusion
This article showed how to create a highly available Red Hat OpenShift cluster on LPAR on IBM Z. The cluster can be used as a base for running both containerized workloads as well as virtual machines with Red Hat OpenShift Virtualization (RHOCPV). The second part of this series explains how to install and use RHOCPV.
Additional Resources