Original Author: Ashwin Hendre

Installer Provisioned Infrastructure method can be used to deploy a private OpenShift cluster on IBM PowerVS.

In scope

This blog covers creating and destroying a “private” OpenShift cluster i.e., one which is accessible only through an internal network and the nodes of the cluster can access the internet. In other words, no public ingress is configured, but outbound networking traffic is permitted.

Out of scope

It is also possible to create an OpenShift cluster using IPI on Power VS for 

1. General scenario: A cluster where DNS, Ingress Controller, and API server are public and the cluster is accessible through the internet. https://community.ibm.com/community/user/powerdeveloper/blogs/ashwin-hendre/2023/07/10/powervs-ipi covers this scenario. 
2. Disconnected scenario: A cluster where the nodes do not have inbound or outbound access to the internet. 

These two scenarios are outside the scope of this blog post.

Requirements for installing a Private Cluster with OpenShift on Power VS with IPI 

1. Create an IBM Cloud account
2. Create an IBM Cloud service ID (optional)
   Refer https://cloud.ibm.com/docs/iam?topic=iam-serviceids&interface=ui#create_serviceid for details on creating it.
3. Create a resource group (optional) 
   Resource group is a logical grouping of resources such that a set of users can be allowed access to that group rather than enabling access to individual resources. 
   Refer https://cloud.ibm.com/docs/account?topic=account-rgs for steps on creating a resource group.
   It is possible that your account admin set a different resource group as the default. Verify the name of the default resource group before continuing. 
4. Create an access group (optional)
   An access group allows you to group users and assign required access policies to them based on their role. Refer https://cloud.ibm.com/docs/account?topic=account-groups for details on creating an access group. 
   Refer https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/installing_on_ibm_power_virtual_server/installing-ibm-cloud-account-power-vs#pre-requisite-permissions-ibm-cloud_installing-ibm-cloud-account-power-vs for information on roles and their respective permissions. 
5. Assign permissions to the access group created as a part of the previous step through either of the following approaches (optional) 
   1. UI (recommended)
      Follow these steps: https://cloud.ibm.com/docs/account?topic=account-groups&interface=ui#access_ag
   2. CLI 
      ibmcloud iam access-group-policy-create GROUP_NAME {-f, --file @JSON_FILE | --roles ROLE_NAME1,ROLE_NAME2... [--service-name SERVICE_NAME] [--service-instance SERVICE_INSTANCE] [--region REGION] [--resource-type RESOURCE_TYPE] [--resource RESOURCE] [--resource-group-name RESOURCE_GROUP_NAME] [--resource-group-id RESOURCE_GROUP_ID]}
6. Create or reuse an API key associated with the service id
   Refer https://cloud.ibm.com/docs/account?topic=account-serviceidapikeys 
7. Create an SSH key pair locally.
8. Create or reuse an IBM Cloud Virtual Private Cloud (VPC). Refer https://cloud.ibm.com/docs/vpc?topic=vpc-getting-started#create-and-configure-vpc for instructions on creating an IBM Cloud VPC. 
   
   1. Create or reuse subnets that don’t allow public access. Make sure to check the subnets’ security group rules to ensure no inbound endpoints are enabled.
   2. Note: If the cluster has any public subnets, load balancer services created by administrators might be publicly accessible. To ensure cluster security, verify that these services are explicitly annotated as private.
9. Create or reuse an IBM Cloud DNS® Service.
   1. Create a DNS zone that matches your cluster’s basedomain. The basedomain is required to be entered while generating the install configuration as a part of Installation step 2.
10. Obtain the openshift-install and oc binaries.
    1. Access the PowerVS IPI installer page on the OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.
    2. Download the OpenShift installer and the Command Line interface that correspond with your host operating system and architecture. Make sure you download the openshift-install and oc archives for the architecture of the system you're using for deployment. For example, if you're using an x86_64 VM to carry out deployment, the installer package may be named openshift-install-linux-amd64-4.22.x.tar.gz.
    3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:  
       tar -xvf openshift-install-linux-amd64-4.22.8.tar.gz && tar -xvf openshift-client-linux-amd64-4.22.8.tar.gz
    4. Download your installation pull secret from the Red Hat OpenShift Cluster Manager. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.
11. Move the openshift-install and oc binaries to a folder in your $PATH, such as /usr/bin/
12. Create or reuse a VM in the that has access to the cluster’s network, the internet and IBM Cloud API. 
    e.g. To create a VM in IBM Cloud, refer https://cloud.ibm.com/docs/vpc?topic=vpc-creating-virtual-servers
13. Generate Cloud Credentials
    Use the ccoctl.sh script by referring to the associated documentation, or follow the manual procedure detailed in steps in 13.1 and 13.4.
    1. Obtain the ccoctl binary. The general approach is as follows. Additional information can be found in this guide.
       1.  Obtain the OpenShift Container Platform release image information by running the following command 
          RELEASE_IMAGE=$(openshift-install version | awk '/release image/ {print $3}')
       2. Obtain the CCO container image info from the OpenShift Container Platform release image by running the following command 
          CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a <path to pull secret file>)
       3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command 
          oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a <path to pull secret file>
       4. Change the permissions to make ccoctl executable by running the following command 
          chmod 775 ccoctl
    2. Extract the credentials
       1. mkdir credreqs
       2. oc adm release extract --cloud=powervs --credentials-requests` $RELEASE_IMAGE --to=./credreqs
       3. ccoctl ibmcloud create-service-id --credentials-requests-dir <path-to-directory-with-list-of-credentials-requests> --name <name> --resource-group-name <resource-group-name>
          resource-group-name is optional but recommended.

Note that cluster resources will be charged as per IBM pricing and follows an operational expenditure model.

Installation steps

1. Create a sub-directory to hold the artifacts the installer creates (install configuration, manifests, metadata, cluster authentication data, etc.)
   1. mkdir ocp-deploy
2. Create the install configuration
   1. openshift-install create install-config --dir=ocp-deploy
       The folder specified as the --dir value must be the same as the above step.
      You will be prompted for several pieces of information required for deploying the cluster
      1. SSH public key
      2. Platform: Select powervs
      3. IBM Cloud User ID
      4. IBM Cloud API Key
      5. Region
         Region where the cluster will be deployed
      6. Zone
         One or more zones contained within the region specified in Step v above. 
      7. Base domain
         A domain created using IBM DNS Service. Domains supported for a private deployment will have (Internal) after them.
         Note: It is necessary to select a base domain with (Internal) after it, while deploying a private cluster. 
      8. Cluster name, a short descriptive name for your cluster
      9. Pull secret
         RedHat pull secret 
   2. If you don’t have an existing workspace or don’t want to use it for cluster deployment, skip steps 3 and 4. The installation program will create a new workspace for you.
   3. Obtain the workspace (formerly called service instance) GUID.
      1. Install IBM cloud CLI by referring https://cloud.ibm.com/docs/cli?topic=cli-install-ibmcloud-cli, selecting the installation method for your operating system
      2. Log in to your IBM cloud account using ibmcloud login
      3. If you’re using Windows, enter ibmcloud resource service-instance <name of the workspace> and note the value in the GUID field of the output
      4. If you’re using Linux or Mac, enter ibmcloud resource service-instance <name of the workspace> | grep -i GUID. Note the id displayed in the output.
   4. Edit install-config.yaml to include the workspace’s GUID.
      1. Open ocp-deploy/install-config.yaml with an editor of your choice.
         e.g. vi ocp-deploy/install-config.yaml 
      2. Add a property platform.powervs.serviceInstanceGUID: GUID of the workspace in which cluster nodes will be created.
   5. If you wish to reuse the created install-config.yaml for future deployments, back it up at a securely as it contains your pull secret. The following steps will delete the install-config.yaml
3. Create manifests
   1. openshift-install create manifests --dir=ocp-deploy
      1. Folder specified must be the same as step 1 above.
   2. Copy the credentials files created in requirement 13 into ocp-deploy/manifests
4. Trigger the installation
   1. openshift-install create cluster --dir=ocp-deploy --log-level=debug
   2. The folder specified must be the same as step 1 above.
   3. The log-level flag is optional but is recommended to be set to debug during manual installation for detailed logs.
5. Post-installation
   1. The output should show how the cluster can be accessed through the CLI and GUI.
   2. The ocp-deploy folder created in step 4 will now contain several useful artifacts, including
      1. A hidden file called .openshift-install.log containing the installation log.
      2. An auth folder containing the kubeconfig to be used to access the cluster through CLI.
      3. A file called metadata.json containing cluster metadata.
         NOTE: This file must be preserved for cluster deletion.
6. A support license can be purchased from try.openshift.com after the cluster is installed.

Destroying an OpenShift cluster

1. Navigate to the folder where the openshift-install binary and the cluster artifact directory are present (here, ocp-deploy).
   1. cd <parent path of openshift-install binary and cluster artifact directory>
2. openshift-install destroy cluster --dir=ocp-deploy --log-level=debug
3. Optional: Delete the Workspace
   1. Log into cloud.ibm.com
   2. Expand the hamburger menu on the top left.
   3. Resource list
   4. Look for the Workspace which was specified during cluster installation and click the three dots next to it.
   5. Delete.
4. Optional: Delete the Cloud credentials
   ccoctl ibmcloud delete-service-id --credentials-requests-dir <path-to-directory-with-list-of-credentials-requests> --name <name>