• Hardware
• Networking
• Kafka
• Elasticsearch
• Flink
• Runtimes
• Storage
• Performance tests

Hardware

Checkpoint

• Verify that the hardware behind is not obsolete.

Procedure

1. Open a Terminal on a Pod or a Worker Node
2. Launch command: cat /proc/cpuinfo
3. You should also verify that your cluster is homogeneous at hardware level
4. The bogomips information might be a good reference to compare your environments

Kubectl command line example

kubectl exec icp4adeploy-odm-decisionserverruntime-69b8d46c77-jqfv4 -n dba2301 -- cat /proc/cpuinfo

Networking

Router

Checkpoint

• Verify that the router configuration is correctly sized. To do this you have to check the number of router pods and verify the CPU consumption of the router's pods.

Procedure

1. In the OpenShift console
2. Open: Home > Search > IngressController > all projects > default > replicas (default was 2)
3. Updated to 5 replicas if needed 

Get the number of replicas for the default Ingress Controller:

kubectl get IngressController default -n openshift-ingress-operator -o=jsonpath='Replicas: {.status.availableReplicas}{"\n"}'

kubectl patch IngressController default -n openshift-ingress-operator --type=json -p '[{ "op": "replace", "path": "/spec/replicas", "value": 5 }]'

HAProxy

Checkpoint

• Verify the HAProxy configuration to ensure that the load is correctly balanced in round-robin and with the right number of nbproc. 
• Of course, here we check the Apache HAProxy as load balancer, but this checkpoint applies to your load balancing solution, but with your procedure though.

Procedure

1. Find the HAProxy node address
2. Use command line: ssh root@haproxy address
3. Look into the haproxy.cfg
1. Use command line: vi /etc/haproxy/haproxy.cfg
2. check ingress-https backend from "balance source" to "balance roundrobin"
3. check nbproc value to 5 (default is 1; here we advise to augment it to 5)
4. Restart HAProxy
1. Use command line: systemctl daemon-reload
2. Use command line: systemctl restart haproxy
3. (optional) systemctl status haproxy

Route Annotation for ADS and ODM runtimes

Checkpoint

• Verify that the load is balanced between every runtime. 
• This annotation is key when the application which calls the decision services is running on a limited set of addresses.

Procedure

1. Open the routes of ADS and ODM
2. Verify that the annotation haproxy.router.openshift.io/balance: roundrobin is well defined in the CPD route

Kubectl command line example

Get Route annotation from the Zen route:

kubectl get Route cpd -n dba2301 -o=jsonpath='Balance: {.metadata.annotations.haproxy\.router\.openshift\.io/balance}{"\n"}'

Upgrade the annotation if needed:

kubectl annotate route cpd -n dba2301 --overwrite haproxy.router.openshift.io/balance='roundrobin'

Kafka 

Checkpoint

• Verify the number of partitions and the retention duration on every Kafka topics used by your BAI.
• The horizontal scalability of BAI depends on Kafka topics configuration.
• The main point is the number of partitions per topic. 
• By default, there is a single partition which limits the performance of BAI and the retention duration is set to one week. 
• In our case, we need 12 partitions, and the retention duration is limited to 2 hours. Here is how we will tell it to OpenShift with an extract of the kafka-topic.yaml:

  apiVersion: ibmevents.ibm.com/v1beta1
  kind: KafkaTopic
  metadata:
    name: icp4ba-bai-ingress
  spec:
    config:
      retention.ms: 7200000
    partitions: 12
    replicas: 3
    topicName: icp4ba-bai-ingress

Procedure

1. Restart Flink Task Managers
1. Delete all the task manager to force a re-initialization 

Kubectl command line example

Get Kafka key information for ODM:

kubectl get KafkaTopic icp4ba-bai-odm-ingress -n dba2301 -o=jsonpath='Partitions: {.spec.partitions}{"\n"}Retention: {.spec.config.retention\.ms}{" ms\n"}'

Get Kafka key information for other products:

kubectl get KafkaTopic icp4ba-bai-ingress -n dba2301 -o=jsonpath='Partitions: {.spec.partitions}{"\n"}Retention: {.spec.config.retention\.ms}{" ms\n"}'

Restart all Flink Task Managers:

for pod in $(kubectl get pods -n dba2301 | grep taskmanager | awk '{print $1}'); do kubectl delete pod $pod -n dba2301 --force --grace-period 0 ; done

Elasticsearch 

Shards

Checkpoint

• When a Time Series inside Elasticsearch contains millions of documents it is mandatory to have several shards.
• The default configuration of BAI is limited to a single shard per Time Series. 

Procedure

1. Add Elasticsearch shards to improve the reliability and performance. 
2. Here is a sample which increases the number of shards to 12. 
   1. Get user: export EK_USER=$(oc get secret iaf-system-elasticsearch-es-default-user -o json  | jq -cr '.data.username' | base64 --decode)
   2. Get password: export EK_PASS=$(oc get secret iaf-system-elasticsearch-es-default-user -o json  | jq -cr '.data.password' | base64 --decode) 
   3. Get the URL: export CLUSTER_ADDRESS=$(oc get routes | grep iaf-system-es | awk '{print $2}')
   4. Verify the version used: curl -ku "$EK_USER:$EK_PASS" https://$CLUSTER_ADDRESS
   5. Change the shards number: curl -vgku "$EK_USER:$EK_PASS" -X POST "https://$CLUSTER_ADDRESS/icp4ba-bai-odm-timeseries-write-ibm-bai/_rollover?pretty" -d'{"aliases":{"icp4ba-bai-odm-timeseries-ibm-bai":{}},"settings":{"index.number_of_shards":12}}' -H 'Content-Type: application/json' 

Additional documentation is available here.

Kubectl command line example

Get the username for ElasticSearch:

kubectl get secret iaf-system-elasticsearch-es-default-user -n dba2301 -o=go-template='{{.data.username|base64decode}}{{"\n"}}'

Get the password for ElasticSearch:

kubectl get secret iaf-system-elasticsearch-es-default-user -n dba2301 -o=go-template='{{.data.password|base64decode}}{{"\n"}}'

Get the route information for ElasticSearch:

kubectl get routes -n dba2301 | grep iaf-system-es | awk '{print $2}'

Memory configuration 

Checkpoint

• The Elasticsearch pod created by IAF contains a request memory of 5Gb.
• The heap size used by Elasticsearch is by default at 1Gb. It should be smaller than the request memory of the pod.
• In our case the request memory is at 5Gb so we should modify The Elasticsearch to use 4G of heap

Procedure

1. Use the open Shift console and access to the stateful sets
2. Add in stateful Set of Elasticsearch Data the following environment variable
1.  ES_JAVA_OPTS -Xms4g -Xmx4g

Kubectl command line example

Adding an additional env property in ElasticSearch (unless the iaf-system change it again):

kubectl patch StatefulSets iaf-system-elasticsearch-es-data -n dba2301 --type=json -p '[{ "op": "add", "path": "/spec/template/spec/containers/0/env/0", "value": {name: "ES_JAVA_OPTS", value: "-Xms4g -Xmx4g"} }]'

Flink

Procedure

1.  Change the Custom Resource of CP4BA following the documentation here.
2. i.e. For a route to the Flink web interface to be automatically created, set the spec.bai_configuration.flink.create_route parameter to true in the custom resource.
3. The user and password to use to access the Web UI of Flink are available in a secret which contains bai-event and admin-user 
4. With this UI you can see the status of the task managers

kubectl patch ICP4AClusters dba2301bai -n dba2301 --type=merge -p '{"spec":{"bai_configuration":{"flink":{"create_route": true}}}}'

kubectl get routes -n dba2301 | grep iaf-system-es | awk '{print $2}'kubectl get InsightsEngine dba2301bai -n dba2301 -o jsonpath='{.status.components.flinkUi.endpoints[?(@.scope=="External")].authentication.secret.secretName}{"\}'

kubectl get secret dba2301bai-insights-engine-flink-admin-user -n dba2301 -o=go-template='{{.data.username|base64decode}}{{"\n"}}'

kubectl get secret dba2301bai-insights-engine-flink-admin-user -n dba2301 -o=go-template='{{.data.password|base64decode}}{{"\n"}}'

kubectl get routes -n dba2301 | grep iaf-system-es | awk '{print $2}'kubectl get InsightsEngine dba2301bai -n dba2301 -o jsonpath='{.status.components.flinkUi.endpoints[?(@.scope=="External")].uri}{"\}'

Runtimes

Checkpoint

• Verify that CPU/memory requests AND limits for ODM and ADS runtimes are equals

Procedure

1. This is set inside the CR 

Kubectl command line example

Consult the actual Decision Server Runtime configuration:

kubectl get ICP4AClusters dba2301bai -n dba2301 -o=jsonpath='Decision Server Runtime Config: {.spec.odm_configuration.decisionServerRuntime}{"\n"}'

Consult the actual Decision Server Console configuration:

kubectl get ICP4AClusters dba2301bai -n dba2301 -o=jsonpath='Decision Server Console Config: {.spec.odm_configuration.decisionServerConsole}{"\n"}'

Consult the actual Decision Center configuration:

kubectl get ICP4AClusters dba2301bai -n dba2301 -o=jsonpath='Decision Center Console Config: {.spec.odm_configuration.decisionCenter}{"\n"}'

Consult the actual Decision Runner configuration:

kubectl get ICP4AClusters dba2301bai -n dba2301 -o=jsonpath='Decision Runner Config: {.spec.odm_configuration.decisionRunner}{"\n"}'

Same patterns apply for ADS.

Storage

Performance tests

JMeter

Checkpoint

• Verify JMeter process usage (CPU/RAM)
• CPU should not reach 80%
• Check the level of network usage should not reach 75%.

Procedure

1. Run "top" or "htop" for cpu/ram usage when JMeter is running to control that the limit reached is not on the JMeter side: you do not have to reach 100% of your CPU limit.
2. If you want to test your application without having the overhead of the network (session creation and authentication), prefer using:
   1. The Basic Authentication method
   2. The "Same user on each iteration" option checked on the Thread group in combination with an HTTP Cookie Manager
   3. The "Use KeepAlive" option checked on the HTTP Request

Latency

Checkpoint

• Verify the latency between the machine where JMeter is executed and the OpenShift cluster under test

Procedure

1. ping the cluster from the bench machine
2. check that the average round trip is under 50ms.

Take Away

#CloudPakforBusinessAutomation#migration#OperationalDecisionManager(ODM)