Hosted Cluster or HCP(Hosted Control Plane) is a concept in OpenShift where an OpenShift cluster is created by hosting control plane components as pods in management cluster in a namespace. It's managed by the project called Hypershift. More about it here.

Where agent is one of the type of hosted cluster deployment where an ISO is used to bringup and attach workers to the hosted cluster. More about it here.

In this blog will try to give discuss on how did we bringup agent based hosted cluster with power worker nodes.

Prerequisite:

• Admin access to an OpenShift cluster (version 4.14+) specified by the KUBECONFIG environment variable.
• The OpenShift CLI (oc) or Kubernetes CLI (kubectl).
• A valid pull secret file for the quay.io/openshift-release-dev repository.

Setup the management cluster:

MCE(Multi Cluster Engine) operator from operator hub would be used to setup the management cluster with required CRDs and operators needed to create the hosted cluster.

Installing MCE:

MCE can be installed via both UI and CLI. Follow below steps to install it via CLI

Create a namespace to install all the resources related to MCE operator

• oc create namespace multicluster-engine

Make that namespace as default namespace 

• oc project multicluster-engine

Create operator group

• cat <<EOF | oc create -f -
  apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    name: multicluster-engine-og
  spec:
    targetNamespaces:
    - multicluster-engine
  EOF

Create multicluster-engine operator subscription, make sure version stable-2.4+ is used.

• cat <<EOF | oc create -f -
  apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    name: multicluster-engine
  spec:
    sourceNamespace: openshift-marketplace
    source: redhat-operators
    channel: stable-2.4
    installPlanApproval: Automatic
    name: multicluster-engine
  EOF

Ensure subscription installs necessary CRDs and components

• oc wait subscription multicluster-engine --for=condition=CatalogSourcesUnhealthy=False --timeout=5m

Create mce instance to enable and install required components

• cat <<EOF | oc create -f -
  apiVersion: multicluster.openshift.io/v1
  kind: MultiClusterEngine
  metadata:
    name: multiclusterengine
    spec:
      availabilityConfig: High
      overrides:
        components:
        - enabled: true
          name: assisted-service
        - enabled: true
          name: cluster-lifecycle
        - enabled: true
          name: cluster-manager
        - enabled: true
          name: discovery
        - enabled: true
          name: hive
        - enabled: true
          name: server-foundation
        - enabled: true
          name: cluster-proxy-addon
        - enabled: true
          name: local-cluster
        - enabled: true
          name: hypershift-local-hosting
        - enabled: true
          name: hypershift
        - enabled: false
          name: managedserviceaccount
        - enabled: true
          name: console-mce
      targetNamespace: multicluster-engine
  EOF

 It may take some time to install all the components. Below command can be used to get status of multiclusterengine created in last step. Wait till it reaches Available status

• oc wait MultiClusterEngine multiclusterengine --for=condition=Available --timeout=10m

Creating AgentServiceConfig:

AgentServiceConfig is required to define the set of OS images for assisted service to use and generate the minimal discovery ISO.

• export STORAGE_CLASS=$(oc get sc | grep default | awk '{print $1}') // If management cluster contains multi zone nodes, need to make sure this storage class’s VOLUMEBINDINGMODE is set to WaitForFirstConsumer. If default storage class does not have this volume binding mode, please create one and use it.  
  export DB_VOLUME_SIZE="10Gi"
  export FS_VOLUME_SIZE="10Gi"
  export OCP_VERSION="4.14.0"
  export OCP_MAJOR=${OCP_VERSION%.*}
  export ARCH="ppc64le"
  export OCP_RELEASE_VERSION=$(curl -s https://mirror.openshift.com/pub/openshift-v4/${ARCH}/clients/ocp/${OCP_VERSION}/release.txt | awk '/machine-os / { print $2 }')
  export ISO_URL="https://mirror.openshift.com/pub/openshift-v4/${ARCH}/dependencies/rhcos/${OCP_MAJOR}/${OCP_VERSION}/rhcos-${OCP_VERSION}-${ARCH}-live.${ARCH}.iso"
  export ROOT_FS_URL="https://mirror.openshift.com/pub/openshift-v4/${ARCH}/dependencies/rhcos/${OCP_MAJOR}/${OCP_VERSION}/rhcos-${OCP_VERSION}-${ARCH}-live-rootfs.${ARCH}.img"

• envsubst <<"EOF" | oc apply -f -
  apiVersion: agent-install.openshift.io/v1beta1
  kind: AgentServiceConfig
  metadata:
   name: agent
  spec:
    databaseStorage:
      storageClassName: ${STORAGE_CLASS}
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: ${DB_VOLUME_SIZE}
    filesystemStorage:
      storageClassName: ${STORAGE_CLASS}
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: ${FS_VOLUME_SIZE}
    osImages:
      - openshiftVersion: "${OCP_VERSION}"
        version: "${OCP_RELEASE_VERSION}"
        url: "${ISO_URL}"
        rootFSUrl: "${ROOT_FS_URL}"
        cpuArchitecture: "${ARCH}"
  EOF

Create Hosted Control Plane:
Build Hypershift binary:

• git clone https://github.com/openshift/hypershift.git
  cd hypershift
  make build
  export PATH=$PATH:$PWD/bin/hypershift

Create Cluster:

• export CLUSTERS_NAMESPACE="clusters"
  export HOSTED_CLUSTER_NAME="agent-test"
  export HOSTED_CONTROL_PLANE_NAMESPACE="${CLUSTERS_NAMESPACE}-${HOSTED_CLUSTER_NAME}"
  export BASEDOMAIN=<your DNS domain>
  export PULL_SECRET_FILE=<path to the pull secret>
  export OCP_RELEASE="4.14.4-multi"
  # Typically the namespace is created by the hypershift-operator
  # but agent cluster creation generates a capi-provider role that
  # needs the namespace to already exist
  oc create ns ${HOSTED_CONTROL_PLANE_NAMESPACE}
  
  hypershift create cluster agent \
      --name=${HOSTED_CLUSTER_NAME} \
      --pull-secret="${PULL_SECRET_FILE}" \
      --agent-namespace=${HOSTED_CONTROL_PLANE_NAMESPACE} \
      --base-domain=${BASEDOMAIN} \
      --ssh-key ${HOME}/.ssh/id_rsa.pub \
      --release-image=quay.io/openshift-release-dev/ocp-release:${OCP_RELEASE} --render > cluster-agent.yaml

Once cluster-agent.yaml is generated, modify the HostedCluster spec's service field with suitable servicePublishingStrategy for various control plane services. By default it is generated with NodePort, you can change it to Route or LoadBalancer based on your requirement.

Sample services block of a HostedCluster's spec will look like below.

• services: 
   - service: APIServer
      servicePublishingStrategy:
        type: LoadBalancer
    - service: OAuthServer
      servicePublishingStrategy:
        type: Route
    - service: OIDC
      servicePublishingStrategy:
        type: None
    - service: Konnectivity
      servicePublishingStrategy:
        type: Route
    - service: Ignition
      servicePublishingStrategy:
        type: Route
    - service: OVNSbDb
      servicePublishingStrategy:
        type: Route

Once you done editing cluster-agent.yaml, you can apply it to deploy the HostedCluster

• oc apply -f cluster-agent.yaml

 Wait for the HostedCluster to be available

• oc wait --timeout=15m --for=condition=Available --namespace=${CLUSTERS_NAMESPACE} hostedcluster/${HOSTED_CLUSTER_NAME}

Create InfraEnv:

• export SSH_PUB_KEY=$(cat <ssh_pub_key path>)
  export ARCH="ppc64le"
  
  envsubst <<"EOF" | oc apply -f -
  apiVersion: agent-install.openshift.io/v1beta1
  kind: InfraEnv
  metadata:
    name: ${HOSTED_CLUSTER_NAME}
    namespace: ${HOSTED_CONTROL_PLANE_NAMESPACE}
  spec:
    cpuArchitecture: $ARCH
    pullSecretRef:
      name: pull-secret
    sshAuthorizedKey: ${SSH_PUB_KEY}
  EOF

Once InfraEnv is created, a minimal iso would be generated by assisted service need to use this iso to bring up the worker nodes. Get the ISO Download URL from below command

• oc -n ${HOSTED_CONTROL_PLANE_NAMESPACE} get InfraEnv ${HOSTED_CLUSTER_NAME} -ojsonpath="{.status.isoDownloadURL}"

Create DNS records:

Create *.api and *.api-int DNS records from the endpoint provided by kube-apiserver service from control plane namespace.

• oc get svc kube-apiserver -n ${HOSTED_CONTROL_PLANE_NAMESPACE}
  
  # Create "*.api.<HOSTED_CLUSTER_NAME>" DNS record points to kube-apiserver endpoint
  # Create "*.api-int.<HOSTED_CLUSTER_NAME>" DNS record points to kube-apiserver endpoint
  

Create *.apps DNS record points to one of the worker's IP or if there is a load balancer on top of the workers, use that to create this DNS record.

• # Create "*.apps.<HOSTED_CLUSTER_NAME>" DNS record points to one of the worker's IP or LB hostname.
  

• Creating the DNS records is a  prerequisite before booting the worker nodes with the discovery ISO.

Boot the worker:

Boot the ISO generated in previous step on a Power node via PowerVC or IBM Cloud PowerVS VSI. Once it's booted, it will appear as an agent in control plane namespace.

Add agents:

Use below command to list the agents in control plane namespace

• oc get agents -n ${HOSTED_CONTROL_PLANE_NAMESPACE} -o=wide

Approve the agent using below command

• oc -n ${HOSTED_CONTROL_PLANE_NAMESPACE} patch agent <agent_id> -p '{"spec":{"approved":true, "hostname": "${HOSTED_CLUSTER_NAME}-worker"}}' --type merge

Once agent is approved, scale the node pool according to the approved agent count

• oc -n ${CLUSTERS_NAMESPACE} scale nodepool ${HOSTED_CLUSTER_NAME} --replicas 1

Confirm cluster is working fine:

Access the hosted cluster via kubeconfig

• hypershift create kubeconfig --namespace ${CLUSTERS_NAMESPACE} --name ${HOSTED_CLUSTER_NAME} > ${HOSTED_CLUSTER_NAME}.kubeconfig
  

Use following commands to validate the agent cluster created

• export KUBECONFIG=${HOSTED_CLUSTER_NAME}.kubeconfig
  
  oc get nodes -o=wide
  NAME                STATUS   ROLES    AGE   VERSION           INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                  CONTAINER-RUNTIME
  agent-test-worker   Ready    worker   13d   v1.27.6+b49f9d1   192.168.140.135   <none>        Red Hat Enterprise Linux CoreOS 414.92.202311082213-0 (Plow)   5.14.0-284.40.1.el9_2.ppc64le   cri-o://1.27.1-13.1.rhaos4.14.git956c5f7.el9
  
  oc get clusterversion
  NAME      VERSION                                    AVAILABLE   PROGRESSING   SINCE   STATUS
  version   4.14.0-0.nightly-multi-2023-11-14-130514   True        False         13d     Cluster version is 4.14.0-0.nightly-multi-2023-11-14-130514

#Containers,Kubernetes,andOpenShift