Hosted Cluster on Power using MCE-Hypershift None Platform

View Only

Hosted Cluster on Power using MCE-Hypershift None Platform

By Dharaneeshwaran Ravichandran posted Mon March 27, 2023 05:43 AM

Like

Introduction to MCE

You can read about MCE here

Introduction to Hypershift

You can read about Hypershift here

Prerequisites:

Management OCP cluster to host your Hosted cluster. Typically ROKS from IBM Cloud can be used as a management cluster. Need to expose API server in public so that the workers residing in private network can talk to routes and hosted cluster's APIs using public ip.
Pull-secret (available at cloud.redhat.com)
SSH public key already available (it can be created as ssh-keygen -t rsa -f /tmp/sshkey -q -N "")
PowerVC instance with RHCOS image imported. If RHCOS image not available, please import the RHCOS image respect to the release that you are going to use to create the Hosted Cluster
Any HTTP server available to host a ignition file (text)
DHCP server to allocate IP addresses to the nodes
DNS server to resolve traffic to the ingresses pod (*.apps.${cluster}.${domain}, should point to a Load Balancer when using more than 1 node)

Steps to install:

Install Hypershift using MCE:

- Use management cluster mentioned in prerequisite for below operations.
- Install MCE operator by following this guide. The simplest option would be to install from Operator Hub from web console.
- As part of MCE operator installation, you might have created MultiClusterEngine Custom Resource, which contains the configuration for MCE installation. Need to enable hypershift-preview component("enabled": "true") of MCE instance created like here
- Once both steps are completed MCE operator, Hypershift operator and necessary CRDs might have got installed in management cluster.

Create None Hosted Cluster:

Import management cluster's kubeconfig to current cli session before proceeding to below steps.
Install Hypershift binary

git clone https://github.com/openshift/hypershift.git
cd hypershift
make build
sudo install -m 0755 bin/hypershift /usr/local/bin/hypershift

Create None type hypershift cluster

#!/usr/bin/env bash

export HOSTED_CLUSTER_NAME="hosted-0"

# Configure DNS server mentioned in prerequisite section to serve this DOMAIN which can be used for api and ingress purposes.
export BASEDOMAIN="example.com"

export PULL_SECRET_FILE=${HOME}/.hypershift/pull_secret.txt
export OCP_RELEASE=4.13.0-ec.3-multi

hypershift create cluster none \
    --name ${HOSTED_CLUSTER_NAME} \
    --pull-secret "${PULL_SECRET_FILE}" \
    --base-domain ${BASEDOMAIN} \
    --ssh-key ${HOME}/.ssh/id_rsa.pub \
    --release-image quay.io/openshift-release-dev/ocp-release:${OCP_RELEASE} --render > cluster-none.yaml

Modify the rendered file - Change the servicePublishingStrategy to LoadBalancer and Route(because ROKS cluster is deployed in the cloud environment and nodes are in private network and can't be accessed by the workers directly).

  - service: APIServer
    servicePublishingStrategy:
      nodePort:
        address: 10.244.0.13
      type: NodePort
  - service: OAuthServer
    servicePublishingStrategy:
      nodePort:
        address: 10.244.0.13
      type: NodePort
  - service: OIDC
    servicePublishingStrategy:
      nodePort:
        address: 10.244.0.13
      type: None
  - service: Konnectivity
    servicePublishingStrategy:
      nodePort:
        address: 10.244.0.13
      type: NodePort
  - service: Ignition
    servicePublishingStrategy:
      nodePort:
        address: 10.244.0.13
      type: NodePort
  - service: OVNSbDb
    servicePublishingStrategy:
      nodePort:
        address: 10.244.0.13
      type: NodePort

to

  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: OIDC
    servicePublishingStrategy:
      type: None
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: OVNSbDb
    servicePublishingStrategy:
      type: Route

Deploy rendered yaml
```
$ oc apply -f cluster-none.yaml
```
After a while, a number of pods will be created in the "cluster-${HOSTED_CLUSTER_NAME}" namespace. Those pods are the control plane of the hosted cluster.

$ oc get pods -n clusters-hosted0                                                    
NAME                                                  READY   STATUS    RESTARTS        AGE
catalog-operator-c5b458cdb-tq5fv                      2/2     Running   0               5m50s
certified-operators-catalog-6fd5f7c794-x7kjb          1/1     Running   1 (2m9s ago)    5m50s
cluster-api-65d859c5dc-zg4fh                          1/1     Running   0               8m28s
cluster-autoscaler-7d9fcbfd4-p94wn                    1/1     Running   0               7m45s
cluster-image-registry-operator-596c5485bf-sx5qj      1/2     Running   0               5m49s
cluster-network-operator-84c7588485-7qrw7             1/1     Running   0               5m52s
cluster-node-tuning-operator-6c4d4f498b-lm9vk         1/1     Running   0               5m52s
cluster-policy-controller-6d489899d4-dw5vr            1/1     Running   0               5m52s
cluster-version-operator-776bffdd8-vmktv              1/1     Running   0               5m52s
community-operators-catalog-557dd754db-lvmk5          1/1     Running   0               5m50s
control-plane-operator-54cdfcdf96-w6z9h               1/1     Running   0               8m27s
dns-operator-6bdb76bbb-6mp6m                          1/1     Running   0               5m51s
etcd-0                                                1/1     Running   0               7m46s
hosted-cluster-config-operator-7b5b4dd6b7-t7mpj       1/1     Running   0               5m50s
ignition-server-7f6c9c7cbb-24r4v                      1/1     Running   0               7m44s
ingress-operator-6999767655-glhk5                     1/2     Running   0               5m51s
konnectivity-agent-56dcffc69c-mtrww                   1/1     Running   0               7m46s
konnectivity-server-577f8d89f9-mxzr7                  1/1     Running   0               7m46s
kube-apiserver-7c998dd574-2nvxx                       3/3     Running   0               7m45s
kube-controller-manager-8478b5489c-c25fp              1/1     Running   0               73s
kube-scheduler-59b99d7ffb-z5fnc                       1/1     Running   0               6m14s
machine-approver-6d567476cc-62fmr                     1/1     Running   0               7m45s
oauth-openshift-68c4cb88c9-2pdqz                      2/2     Running   0               65s
olm-operator-ffb65c4f7-w2nz9                          2/2     Running   0               5m49s
openshift-apiserver-84f6c7c964-mqzqs                  2/2     Running   0               73s
openshift-controller-manager-5bd79757f-dpgfz          1/1     Running   0               5m53s
openshift-oauth-apiserver-6b9557c5d7-fq2jd            1/1     Running   1 (2m18s ago)   5m53s
openshift-route-controller-manager-6f9bc587cc-5nlwv   1/1     Running   0               5m52s
packageserver-f7758dc64-ss9wj                         2/2     Running   0               5m49s
redhat-marketplace-catalog-7445f9bb8-67cx6            1/1     Running   0               5m50s
redhat-operators-catalog-6f6fc8cb65-c4j7w             1/1     Running   0               5m50s

Download the ignition generated in the Hosted cluster. Once all the HostedCluster pods are Running you will be able to download the ignition file.

$ IGNITION_ENDPOINT=$(oc get hc ${HOSTED_CLUSTER_NAME} -n clusters -o json | jq -r '.status.ignitionEndpoint')
$ IGNITION_TOKEN_SECRET=$(oc -n clusters-${HOSTED_CLUSTER_NAME} get secret | grep token-${HOSTED_CLUSTER_NAME}  | awk '{print $1}')
$ set +x
$ IGNITION_TOKEN=$(oc -n clusters-${HOSTED_CLUSTER_NAME} get secret ${IGNITION_TOKEN_SECRET} -o jsonpath={.data.token})
$ curl -s -k -H "Authorization: Bearer ${IGNITION_TOKEN}" https://${IGNITION_ENDPOINT}/ignition > worker.ign

Need to host this worker.ign generated in last step in a http server so that while creating VM in PowerVC user-data can be passed with ignition configuration.
Create a VM in PowerVC with the RHCOS image mentioned in prerequisite section with the ignition content created above.
Once VM created add the ip entry in DNS config and restart named service.
Once worker successfully created, you would be able to see its getting listed in Hosted cluster. For that download the Hosted cluster's kubeconfig and import it in a cli session.

$ hypershift create kubeconfig --name ${HOSTED_CLUSTER_NAME} > ${HOSTED_CLUSTER_NAME}-kubeconfig

After a while, the worker will be listed. Before that use below command to approve the CSR.

$ oc get csr --kubeconfig=${HOSTED_CLUSTER_NAME}-kubeconfig -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve --kubeconfig=${HOSTED_CLUSTER_NAME}-kubeconfig

You may need to repeat above step until all CSR gets approved.
Once all the CSR were approved, you would start seeing node getting listed. Verify the worker nodes and cluster operators like below.

$ oc get nodes --kubeconfig=${HOSTED_CLUSTER_NAME}-kubeconfig
NAME                    STATUS   ROLES    AGE   VERSION
localhost.localdomain   Ready    worker   40m   v1.25.2+5533733

$ oc get co --kubeconfig=${HOSTED_CLUSTER_NAME}-kubeconfig
NAME                                       VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    4.12.0-rc.2   True        False         False      41m
csi-snapshot-controller                    4.12.0-rc.2   True        False         False      44m
dns                                        4.12.0-rc.2   True        False         False      44m
image-registry                             4.12.0-rc.2   True        False         False      44m
ingress                                    4.12.0-rc.2   True        False         True       154m    The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
insights                                   4.12.0-rc.2   True        False         False      45m
kube-apiserver                             4.12.0-rc.2   True        False         False      155m
kube-controller-manager                    4.12.0-rc.2   True        False         False      155m
kube-scheduler                             4.12.0-rc.2   True        False         False      155m
kube-storage-version-migrator              4.12.0-rc.2   True        False         False      44m
monitoring                                 4.12.0-rc.2   True        False         False      42m
network                                    4.12.0-rc.2   True        False         False      45m
node-tuning                                4.12.0-rc.2   True        False         False      48m
openshift-apiserver                        4.12.0-rc.2   True        False         False      155m
openshift-controller-manager               4.12.0-rc.2   True        False         False      155m
openshift-samples                          4.12.0-rc.2   True        False         False      43m
operator-lifecycle-manager                 4.12.0-rc.2   True        False         False      154m
operator-lifecycle-manager-catalog         4.12.0-rc.2   True        False         False      155m
operator-lifecycle-manager-packageserver   4.12.0-rc.2   True        False         False      155m
service-ca                                 4.12.0-rc.2   True        False         False      45m
storage                                    4.12.0-rc.2   True        False         False      45m

Power Developer eXchange

A place for IBM Power open source developers to learn,
collaborate, contribute, and create

Containers, Kubernetes, OpenShift on Power

Hosted Cluster on Power using MCE-Hypershift None Platform

By Dharaneeshwaran Ravichandran posted Mon March 27, 2023 05:43 AM

Permalink

Power Developer eXchange

A place for IBM Power open source developers to learn, collaborate, contribute, and create

Containers, Kubernetes, OpenShift on Power

Hosted Cluster on Power using MCE-Hypershift None Platform

By Dharaneeshwaran Ravichandran posted Mon March 27, 2023 05:43 AM

Permalink

A place for IBM Power open source developers to learn,
collaborate, contribute, and create