Thanks for the answer Ross.
We know how to configure and tune ODM XU, we have been trying different settings of pool size, but eventually, during short time periods we have extremely high response times from ODM and all the client apps start receiving timeouts. This why I was wondering if maybe some other configuration parameter in liberty may help.
In our case, we have 10-11 millions request per day for around 60 decision services deployed in ODM. There are 2 nodesx3cores balanced. Some services are complex and take around 150 msec average. But most of them are very fast (10-20 msec average).
But when the performance degrades, even a very simple service decision service with average response time of 1 msec start giving response times over 10 sec meassured in the acces.log of liberty (that is, accounting execution time in the XU + reading the request + wirting the request + whatever). However, in the RES console, the max execution time rarely goes over 100 msec in this period.
The max pool size is 250 in each node, we have a timeout of 1sg waiting for an engine in the pool, so the problem is not that request is waiting for the pool. Time should be in some other point. CPU average is ok, and has some peaks, but even with peaks is very rare for me passing from 1 msec up to 10 sec. It should be something more.
This is the reason why I was asking in this community
------------------------------
Eduardo Izquierdo Lázaro
Automation Architect
DECIDE
Madrid
609893677
------------------------------
Original Message:
Sent: Thu January 25, 2024 09:01 AM
From: Ross MacDougall
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
For ODM on Liberty, XU Connection Pool sizing is done by adjusting the maxPoolSize in the ra.xml in the DecisionService.war file, not in Liberty itself. The default value is 20. I typically externalize the ra.xml by extracting the file from the war, and pre-pending a new directory onto the CLASSPATH in the jvm.options file. I can then modify the ra.xml without repacking the .war.
Step 1. Create a directory in the server home directory
cd /opt/IBM/wlp/usr/servers/myServer
mkdir ./res
Step 2. Add the following to the jvm.options
-Xbootclasspath/p:./res
Step 3. Extract the ra.xml from the DecisionService.war file and place it into the ./res
I have a directive in my server.xml to auto expand my .war files. This helps with performance
cp ./apps/expanded/DecisionService.war/WEB-INF/classes/ra.xml ./res/ra.xml
Step 4. Modify ra.xml and restart the server.
Modify the defaultConnectionManagerProperties stanza in the ra.xml
Change the maxpoolsize value to your desired value. As an example
<config-property>
<config-property-name>defaultConnectionManagerProperties</config-property-name>
<config-property-type>java.lang.String</config-property-type>
<config-property-value>
pool.maxSize=80,pool.waitTimeout=-1
</config-property-value>
</config-property>
Restart your server. Verify through your RES console - Server Information. Select Log Level = DEBUG, and dump the XU. You should see new values.
Our Environment
We do 2M ODM invocations per day on a 4 node (4 CPUs per node) WLP config behind a load balancer. 50 ms response times. 400K-600K JSON request payloads.
Other things to look at would be Document Builder Pool and Transformation Pool in the ra.xml. I'd also look at max heap size for ODM in WLP.
------------------------------
Ross MacDougall
Original Message:
Sent: Mon January 22, 2024 08:39 AM
From: Eduardo Izquierdo Lázaro
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
We're running services executed by IBM ODM and the system has a very high throughput (up to 9k request per minute). This is a cluster with 2 nodes x 3 cores balanced by a previous LoadBalancer.
We're having performance issues and we're not sure if the problem is in the configuration of ODM or liberty. We're trying different tunning in ODM but we're not sure abouthe liberty tunning.
What is the default value maxPoolSize of liberty 20.0.0.9?
What would it be a recommended value for this throughput?
Note that some request have a very heavy JSON payload (up to more than 1M). Is there any other parameter we can tune related to the parsing of request payload?
------------------------------
Eduardo Izquierdo Lázaro
Automation Architect
DECIDE
Madrid
609893677
------------------------------