Introduction
Operational Decision Manager (ODM) empowers business users and developers to collaborate when they automate an organization’s business policies. ODM automates the decision-making process and governs future policy updates. Execution of the business rules (decision services at runtime) can scale out in clusters of servers running on bare metal, virtual machines, or containers. In this article we will focus on OpenShift with dedicated Common Services.
These tunings were tested on CP4BA 24.0.1.
The tuning of CP4BA and ODM is key to get the best performance possible. This tuning should be adapted to the SLA and the complexity of your applications.
The CP4BA architecture with Operational Decision Manager
The tuning of CP4BA layers and ODM component is impacting the performance and the SLA of your applications.
Components and Environments
The main components of ODM in a topology are:
- Decision Server console to manage the runtimes
- Decision Server runtime to execute the decisions services
- Decision Center to author the decisions services
- Decision Runner (non-production) to execute tests and simulations on decisions
Gold Topology with two sandboxes and a pre-prod
Why optimize and tune CP4BA and ODM?
- Without any tuning the performance cannot be optimal
- License of ODM/CP4BA not correctly used
- Infrastructure not well balanced implies losses of money
- SLA not always meet
- The performance of ODM in CP4BA depends on CP4BA CRD objects, and configurations like
- CommonService (Cloud Pak foundational services)
- ZenServices (Cloud Pak foundational services)
- Shared configuration (ICP4ACluster configuration)
- ODM configuration (ICP4ACluster configuration)
But also on the design of the Decision Services inside ODM.
General Tuning Guidelines for Production Environment
- The network latency between the Cluster and the database has a huge impact on the Decision Center performance. This latency should be under 5 ms.
- Ephemeral Storage & impact on stability: if the Ephemeral storage is configured too tiny, then the ODM pod will be evicted, and a new pod will start.
Tuning Foundational Services
- Foundational services sizing is available in this documentation :
https://www.ibm.com/docs/en/cloud-paks/foundational-services/4.6.0?topic=installer-hardware-requirements-recommendations-foundational-services
With ODM only the Foundational Services should be set to small like this :
-
- In the OpenShift console:
- Search > Resources ‘CommonService’ > common-service
- In the YAML, change the spec.size value
- starterset
- small
- medium
- large
Tuning Shared Configuration
- With ODM only the Shared Configuration should be set to small.
- With BAI you should set it to medium or large.
- Change it using the Shared Configuration
- In the OpenShift console:
- Search > Resources ‘ICP4ACluster’ > select your CR deployment
- In the YAML, change the spec.shared_configuration.sc_deployment_profile_size value
Tuning Zen Service
- Align the profile (small, medium, large) of your ZenService to the targeted profile of your ODM product.
- In the OpenShift console:
- Search > Resources ‘ZenService’ > common-service
- In the YAML, change the spec.scaleConfig value
- starterset
- small
- medium
- large
Tuning of Authentication
There are three modes of authentication supported by CP4BA and ODM
-
- Basic Authentication
- Bearer Token
- Zen Api Key
The performance of Bearer Token and Zen Api Key depends on the number of nginx pods
If the performance with these authentications should be increased, you should
try to increase the number of nginx pods and set the limit cpu of the pods to 1 cpu.
The Deployment to modify is ibm-nginx
Note: This tuning could be lost with an upgrade of Zen.
Tuning ODM configuration
Adapt profile (small, medium, large) of your ODM configuration to the targeted profile of you ODM product.
- Change it using the ODM Configuration
- In the OpenShift console:
- Search > Resources ‘ICP4ACluster’ > select your CR deployment
- In the YAML, change the spec.odm_configuration.deployment_profile_size value
ODM JVM Tuning on Container
- The JVM tuning is not always optimal and should be updated
- By default, only the max heap size of an ODM component is set to 75% of the memory limit of the container.
- To set an initial heap size you should replace one of the following config map:
- <release_name>-odm-ds-runtime-jvm-options-configmap,
- <release_name>-odm-ds-console-jvm-options-configmap,
- <release_name>-odm-dc-jvm-options-configmap,
- <release_name>-odm-dr-jvm-options-configmap
See: https://www.ibm.com/docs/fr/cloud-paks/cp-biz-automation/24.0.1?topic=manager-customizing-jvm-options
Replace the following options -XX:+UseContainerSupport -XX:MaxRAMPercentage=75
By
-XX:+UseContainerSupport -XX:InitialRAMPercentage=40 -XX:MaxRAMPercentage=75
In that case the initial heap size is 40% of the memory limit of the container and the max heap size is 75% of the memory limit of the container.
This tuning impact depends on the ruleset and on the platform used
- With the latest version of the JVM the default setting of the JIT is not always optimal especially for the Decision Server Runtime.
- You need to add to the JVM options in the config map the following option:
-Xtune:throughput
This tuning has a real impact on the execution performance especially if your ruleset is using the automatic Exception Handling.
This option is set by default for Decision Server Runtime in the default configMap
Tuning Frontend Layer (HA proxy)
- Check the number of thread allocated to your frontend/HAProxy and verify that the configuration is using the roundrobin algorithm
- As an example, edit the /etc/haproxy/haproxy.cfg file and change
- the ingress-https backend from "balance source" to "balance roundrobin“
- the nbthread should be at least set to 5
Performance of Decision Center
The performance of Decision Center depends mainly on two factors Latency and Database Performance.
Latency should be verified at several levels:
- From Web browser to Decision Center
- From Decision Center to Database
- From Database to Disk
Database Performance depends on:
- CPU
- Single threaded performance
- Number of concurrent requests
- Memory
- Disk performance ( low latency and SSD)
Conclusion
These tunings are key to meet the SLA of your applications and of Decision Center. Don’t hesitate to modify your setting to improve the performance of your system but not directly on production without a test.
The performance of a Decision Service depends also on the design of your ruleset inside ODM but that is another story.
There is also some useful resources in the Redhat documentation and ODM documentation:
- OpenShift Scalability and Performance Redhat document
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/pdf/scalability_and_performance/openshift_container_platform-4.18-scalability_and_performance-en-us.pdf
- Tuning Guide of ODM version 9.0
https://community.ibm.com/community/user/viewdocument/tuning-guide-odm-900?CommunityKey=c0005a22-520b-4181-bfad-feffd8bdc022&tab=librarydocuments
#OperationalDecisionManager(ODM) #performance #performancetuning #CP4Automation #CP4BA