IBM Cloud Pak for Business Automation
Performance Check List of OCP for CP4BA 26.0.0
A practical checklist for validating OpenShift infrastructure, networking, runtime sizing, and load-test readiness before moving CP4BA workloads to production.
Why this checklist matters
Moving to production requires to take care of some core performance aspects, so that you avoid troubleshooting afterward.
Here is a list of check points we've gathered when planning to use the Cloud Pak for Business Automation (CP4BA) 26.0.0, with a focus on:
ODM Operational Decision Manager
DICMS Decision Intelligence Client Managed Software
For more details on those components and capabilities, please refer to CP4BA documentation.
This article takes the performance tester point of view and split checkpoints into several categories:
1. Hardware
Hardware age and cluster homogeneity
Checkpoint
Verify that the hardware behind is not obsolete.
Procedure
In the OpenShift console, open a Terminal on a Pod or a Worker Node.
Launch command: cat /proc/cpuinfo.
You should also verify that your cluster is homogeneous at hardware level.
The bogomips information might be a good reference to compare your environments.
Kubectl command-line example
kubectl exec ICP4ADEPLOY-ODM-DECISIONSERVERRUNTIME-69B8D46C77-JQFV4 -n DBA2600 -- cat /proc/cpuinfo
2. Networking
Network performance is key especially with a micro service architecture like the CP4BA one. In this category, we target Router configuration, Route annotations and Load balancing settings.
Router
Checkpoint
Verify that the router configuration is correctly sized. To do this you have to check the number of router pods and verify the CPU consumption of the router pods.
Procedure
In the OpenShift console, open:
Home > Search > IngressController > all projects > default > replicas
Update the value to 5 replicas if needed.
Kubectl command-line example
# Get the number of replicas for the default Ingress Controller
kubectl get IngressController default -n openshift-ingress-operator \
-o=jsonpath='Replicas: {.status.availableReplicas}{"\n"}'
# Increase the number of replicas for the default Ingress Controller
kubectl patch IngressController default -n openshift-ingress-operator \
--type=json \
-p '[{ "op": "replace", "path": "/spec/replicas", "value": 5 }]'
HAProxy
Checkpoint
Verify the HAProxy configuration to ensure that the load is correctly balanced in round-robin and with the right number of nbthread.
Procedure
Of course, here we check the Apache HAProxy as load balancer, but this checkpoint applies to your load balancing solution, and with your procedure though.
- Find the HAProxy node address.
- Connect using SSH.
- Open
/etc/haproxy/haproxy.cfg.
- Check the
ingress-https backend and switch from balance source to balance roundrobin.
- Check the
nbthread value. The default is 1; here we advise to augment it to 5.
- Restart HAProxy.
Command-line example
ssh root@<haproxy-address>
vi /etc/haproxy/haproxy.cfg
systemctl daemon-reload
systemctl restart haproxy
# Optional
systemctl status haproxy
Route Annotation for DICMS and ODM runtimes
Checkpoint
Verify that the load is balanced between every runtime.
This annotation is key when the application which calls the decision services is running on a limited set of addresses.
Procedure
In the OpenShift console, open the routes of CPD route, which is the landing page used to access DICMS and ODM.
Verify that the annotation haproxy.router.openshift.io/balance: roundrobin is well defined in the CPD route.
Kubectl command-line example
# Get Route annotation from the Zen route
kubectl get Route cpd -n dba2600 \
-o=jsonpath='Balance: {.metadata.annotations.haproxy\.router\.openshift\.io/balance}{"\n"}'
# Upgrade the annotation if needed
kubectl annotate route cpd -n dba2600 --overwrite \
haproxy.router.openshift.io/balance='roundrobin'
3. Runtimes
To get best performance for ODM and DICMS runtimes you must consider CPU and memory settings.
Recommendation
We advise to check CPU/memory requests AND limits and align their values.
In your CASE package, go to the relevant folder in cert-kubernetes/descriptors/patterns to find all of the templates:
- For ODM, use
ibm_cp4a_cr_production_FC_decisions.yaml.
- For DICMS, use
ibm_cp4a_cr_production_FC_decisions_ads.yaml.
For more information about downloading your CASE package, see Preparing a client to connect to the cluster.
ODM and DICMS runtime requests and limits
Checkpoint
Verify that CPU/memory requests AND limits for ODM and DICMS runtimes are equal.
Procedure
This is set inside the CR.
Kubectl command-line examples
# Consult the actual Decision Server Runtime configuration
kubectl get ICP4AClusters dba2600bai -n dba2600 \
-o=jsonpath='Decision Server Runtime Config: {.spec.odm_configuration.decisionServerRuntime}{"\n"}'
# Consult that Decision Server Console is running
echo "Decision Server Console Config: "$(kubectl get pod | grep decisionserverconsole | awk '{print $3}')
# Consult the actual Decision Center configuration
kubectl get ICP4AClusters dba2600bai -n dba2600 \
-o=jsonpath='Decision Center Console Config: {.spec.odm_configuration.decisionCenter}{"\n"}'
# Consult the actual Decision Runner configuration
kubectl get ICP4AClusters dba2600bai -n dba2600 \
-o=jsonpath='Decision Runner Config: {.spec.odm_configuration.decisionRunner}{"\n"}'
DICMS
Same patterns apply for DICMS.
4. Performance tests
You might want to verify you have sufficient performance results, and for this, you might already have invested in a performance tool like JMeter.
JMeter is good at "injecting" requests to leverage runtimes.
In this case, you should take care of the JMeter process behavior and its possible network latency that could impact performance results.
JMeter
Checkpoint
- Verify JMeter process usage (CPU/RAM).
- CPU should not reach 80%.
- Check the level of network usage should not reach 75%.
Procedure
From your injector machine, run top or htop for CPU/RAM usage when JMeter is running to control that the limit reached is not on the JMeter side: you do not have to reach 100% of your CPU limit.
If you want to test your application without having the overhead of the network, session creation and authentication, prefer using:
- The Basic Authentication method.
- The Same user on each iteration option checked on the Thread group in combination with an HTTP Cookie Manager.
- The Use KeepAlive option checked on the HTTP Request.
Latency
Checkpoint
Verify the latency between the machine where JMeter is executed and the OpenShift cluster under test.
Procedure
Ping the cluster from the bench machine.
Check that the average round trip is under 50 ms.
5. Take Away
There is a useful troubleshooting documentation: