I've developed the following script to help you get started deploying multiarchitecture applications and show elaborate on the techniques for controllin multiarch compute. This script uses the sock-shop application which is available at https://github.com/ocp-power-demos/sock-shop-demo . This series of instructions for sock-shop-demo requires kustomize and following the readme.md in the repository to setup the username and password for mongodb.
You do not need to do every step that follows, please feel free to install/use what you'd like. I recommend the kustomize install with multi-no-ns, and then playing with the features you find interesting. Note, multi-no-ns requires no namespace.
The layout of the application is described in this diagram:
Deploying a non-multiarch Intel App
This deployment shows the Exec errors and pod scheduling errors that are encountered when scheduling Intel only Pods on Power.
For these steps, you are going to clone the ocp-power-demos's sock-shop-demo and then experiment to resolve errors so the application is up and running.
I'd recommend running this from a bastion.
- Clone the repository
git clone https://github.com/ocp-power-demos/sock-shop-demo
-
Switch to the sock-shop-demo folder
-
Download kustomize - this tool enable a ordered layout of the resources. You'll also need oc installed.
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
Ref: https://kubectl.docs.kubernetes.io/installation/kustomize/binaries/
The reason kustomize is used is due to the sort order feature in the binary.
-
Update the manifests/overlays/single/env.secret file with a username and password for mongodb. openssl rand -hex 10 is a good tip to generating a random password. You'll need to copy this env.secret in each 'overlays/
-
We're going to create the sock-shop application.
❯ kustomize build manifests/overlays/single | oc apply -f -
This create a full application within the OpenShift project.
To see the layout of the application you can see the third diagram of the layout (except these are only Intel images) https://github.com/ocp-power-demos/sock-shop-demo/blob/main/README.md#diagrams
- Run
oc get pods -owide
❯ oc get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
carts-585dc6c878-wq6jg 0/1 Error 6 (2m56s ago) 6m21s 10.129.2.24 mac-01a7-worker-0 <none> <none>
carts-db-78f756b87c-r4pl9 1/1 Running 0 6m19s 10.131.0.32 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
catalogue-77d7c444bb-wnltt 0/1 CrashLoopBackOff 6 (8s ago) 6m17s 10.130.2.21 mac-01a7-worker-1 <none> <none>
catalogue-db-5bc97c6b98-v9rdp 1/1 Running 0 6m16s 10.131.0.33 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 5 (2m44s ago) 6m14s 10.129.2.25 mac-01a7-worker-0 <none> <none>
orders-5dbf8994df-whb9r 0/1 CrashLoopBackOff 5 (2m47s ago) 6m13s 10.130.2.22 mac-01a7-worker-1 <none> <none>
orders-db-7544dc7fd9-w9zh7 1/1 Running 0 6m11s 10.128.3.83 rdr-mac-cust-el-tmwmg-worker-2-5hbxg <none> <none>
payment-6cdff467b9-n2dql 0/1 Error 6 (2m53s ago) 6m10s 10.130.2.23 mac-01a7-worker-1 <none> <none>
queue-master-c9dcf8f87-c8drl 0/1 CrashLoopBackOff 5 (2m41s ago) 6m8s 10.129.2.26 mac-01a7-worker-0 <none> <none>
rabbitmq-54689956b9-rt5fb 2/2 Running 0 6m7s 10.131.0.34 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
session-db-7d4cc56465-dcx9f 1/1 Running 0 6m5s 10.130.2.24 mac-01a7-worker-1 <none> <none>
shipping-5ff5f44465-tbjv7 0/1 Error 6 (2m51s ago) 6m4s 10.130.2.25 mac-01a7-worker-1 <none> <none>
user-64dd65b5b7-49cbd 0/1 CrashLoopBackOff 5 (2m25s ago) 6m3s 10.129.2.27 mac-01a7-worker-0 <none> <none>
user-db-7f864c9f5f-jchf6 1/1 Running 0 6m1s 10.131.0.35 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
You might be lucky enough for the scheduler to assign these to Intel only nodes.
At this point if they are all Running with no restarts, yes it's running.
- Grab the external URL
❯ oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
sock-shop sock-shop-test-user-4.apps.XYZ.net front-end 8079 edge/Redirect None
- Open a Browser, and navigate around. Try registering a user.
It failed for me.
Cordon Power nodes
The purpose is to cordon the Power Nodes and delete the existing pod so you get the Pod running on the architecture you want. This is only recommended on a dev/test system and on the worker nodes.
- Find the Power workers
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
- For each of the Power, cordon the nodes
oc adm cordon node/<worker>
- List the front-end app pods
❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 13 (26s ago) 42m
- Delete the
front-end pods.
oc delete pod/front-end-648fdf6957-bjk9m
The app should be running correctly at this point.
Use a Node Selector for the Application
Demonstrate how to use node selector to put the workload on the right nodes.
These microservices use Deployments. We can modify the deployment to use NodeSelectors.
-
Edit the manifests/overlays/single/09-front-end-dep.yaml or oc edit deployment/front-end
-
Find the nodeSelector field and add an architecture limitation using a Node label:
nodeSelector:
node.openshift.io/os_id: rhcos
kubernetes.io/arch: amd64
-
If you edited, the file run oc apply -f manifests/overlays/single/09-front-end-dep.yaml
-
List the front-end app pods
❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 14 (2m49s ago) 50m
front-end-7bd476764-t974g 0/1 ContainerCreating 0 40s
- It may not be 'Ready', and you may need to delete the
front-end pod on the power node.
oc delete pod/front-end-648fdf6957-bjk9m
Note, you can run the following to run with nodeSelectors.
❯ kustomize build manifests/overlays/single-node-selector | oc delete -f -
❯ kustomize build manifests/overlays/single-node-selector | oc apply -f -
Are the pods running on the Intel node?
Uncordon the Power nodes
With the nodeSelector now started, you can uncordon the Power nodes. This is only recommended on a dev/test system and on the worker nodes.
- Find the Power workers
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
- For each of the Power, uncordon the nodes
oc adm uncordon node/<worker>
- List the front-end app pods
❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-6944957cd6-qmhhg 1/1 Running 0 19s
The application should be running. If not, please use:
❯ kustomize build manifests/overlays/single-node-selector | oc delete -f -
❯ kustomize build manifests/overlays/single-node-selector | oc apply -f -
The workload should be all on the intel side.
Deploying a multiarch Intel/Power App
With many of these applications, there are architecture specific alternatives. You can run without NodeSelectors to get the workload scheduled where there is support.
To switch to Node selectors use across Power/Intel.
-
Switch to oc project sock-shop
-
Delete the Pods and Recreate (this is a manifest-listed set of images)
❯ kustomize build manifests/overlays/multi-no-ns | oc apply -f -
- List the app pods
❯ oc get pods -owide
Update the app deployment to use a manifest listed image and remove node selector
We're going to move one of the applications' dependencies using rabbitmq. The IBM team has created a port of Redis to ppc64le. link
-
Edit git/sock-shop-demo/manifests/overlays/multi-no-ns/19-rabbitmq-dep.yaml
-
Replace image: kbudde/rabbitmq-exporter on line 32. icr.io/ppc64le-oss/rabbitmq-exporter-ppc64le:1.0.0-RC19
-
Remove the nodeSelector label kubernetes.io/arch: amd64 limitation on line 39
-
Build and replace kustomize build manifests/overlays/multi-no-ns | oc apply -f -
-
Check the Pod is starting/running on Power
❯ oc get pod -l name=rabbitmq -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rabbitmq-65c75db8db-9jqbd 2/2 Running 0 96s 10.130.2.31 mac-01a7-worker-1 <none> <none>
The pod should now start on the Power node.
You've taken advantage of the containers, and you can take advantage of other OpenSource container images https://community.ibm.com/community/user/powerdeveloper/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr
Using a Taint/Toleration
Taints and Tolerations provide a way to
- Find the Secondary workers, for instance, if the primary architecture is Intel, you'll get the power workers, and taint the Power workers
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
- Taint the Power Customers
oc adm taint nodes node1 kubernetes.io/arch=ppc64le:NoSchedule
Also note, the taints are flipped (intel is tained with power taint)
-
Edit manifests/overlays/multi-taint-front-end/09-front-end-dep.yaml
-
Update the toleration to match your architecture taint.
-
Save the file
-
Run the oc apply command oc apply -f manifests/overlays/multi-taint-front-end/09-front-end-dep.yaml
-
Check the location of the workers
❯ oc get pods -o wide -l name=front-end
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
front-end-69c64bf86f-98nkc 0/1 Running 0 9s 10.128.3.99 rdr-mac-cust-el-tmwmg-worker-2-5hbxg <none> <none>
front-end-7f4f4844c8-x79zn 1/1 Running 0 103s 10.130.2.33 mac-01a7-worker-1 <none> <none>
You might have to give a few minutes before the workload shifts.
- Check again to see the workload is moved to an intel node:
❯ oc get pods -o wide -l name=front-end
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
front-end-69c64bf86f-98nkc 1/1 Running 0 35s 10.128.3.99 rdr-mac-cust-el-tmwmg-worker-2-5hbxg <none> <none>
Ref: OpenShift 4.14: Understanding taints and tolerations
Summary
These are different techniques to help schedule/control workload placement and help you explore Multi-Arch Compute.
My colleague, Punith, and I have also posted two documents on further controlling workload placement:
- Multi-Arch Compute: Node Selector
- Controlling Pod placement based on weighted node-affininty with your Multi-Arch Compute cluster