If you get thread dumps from the ODM JVMs during the period when responses are very slow, you can check whether threads are waiting for some resource before proceeding.
You can also check whether requests are queuing at the Liberty thread pool waiting for a thread by running with thread pool trace active. The trace spec for that, to be added to Liberty server.xml, is:
<logging consoleFormat="simple" traceSpecification="com.ibm.ws.threading.internal.ThreadPoolController=all=enabled" maxFileSize="1024" maxFiles="10" />
Original Message:
Sent: Wed April 03, 2024 03:21 AM
From: Eduardo Izquierdo Lázaro
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
Hi Sasireka, we think that the problem may be related to the limits at the OS level (if you are in Linux). Ross pointed us in this direction and we discover that servers provisioned by the customer have very low limits. You can check it by running the commands:
ulimit -Sa (for the soft limits)
ulimit -Ha (for the hard limits)
We opened a support case to get some recommended settings and we got this:
------------------------------------------------------
For Linux, we have below technote for recommended limits.
Insufficient ulimit -u (NPROC) Value Causes Native OutOfMemory warning message to be issued
https://www.ibm.com/support/pages/insufficient-ulimit-u-nproc-value-causes-native-outofmemory-warning-message-be-issued
User Limits (in bytes except for NOFILE and NPROC)
type soft limit hard limit
RLIMIT_AS unlimited unlimited
RLIMIT_CORE unlimited unlimited
RLIMIT_CPU unlimited unlimited
RLIMIT_DATA unlimited unlimited
RLIMIT_FSIZE unlimited unlimited
RLIMIT_LOCKS unlimited unlimited
RLIMIT_MEMLOCK 65536 65536
RLIMIT_NOFILE 65536 65536
RLIMIT_NPROC 131072 131072
-----------------------------------------------------------
We don't know yet if this is a solution, because these settings can only be modified by root user, so we asked client to do it, but the bureaucracy is taking long and now we're in freeze period.
If you read the link sent by Brian, it warns about this problem. Normally liberty should manage properly the peaks by increasing/decreasing the threads, but if limited by OS, it can't.
Good luck
------------------------------
Eduardo Izquierdo Lázaro
Automation Architect
DECIDE
Madrid
609893677
Original Message:
Sent: Mon April 01, 2024 03:42 PM
From: Sasireka Dinesh
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
Hello @Eduardo,
Just following up on your question.
Were you able to diagnose the problem and identify the root cause for your performance degradation ?
We are facing a similar situation to handle around 400 req (Soap req) in a second.
Initially, we noticed the following error from XU, which made us bump the pool size to 180 (defaultConnectionManagerProperties, document builder pool and transformation pool). We, no longer see this XU error but the response times are higher.
ilog.rules.res.session.IlrSessionException: An error occurred while the rule session was called: com.ibm.rules.res.xu.internal.XUException: XU client error. com.ibm.rules.res.xu.spi.internal.LocalizedResourceAllocationException: The default connection manager pool is full. ilog.rules.res.xu.pool.IlrPoolException: The pool is full.
On ODM810 nodes (WAS), for the same volume of payload, the average response time is 460 msec but on ODM812 (wlp) its about 10sec.
We have 3 nodes * 4 cores balanced by an ALB.
Any information, that you can share will be helpful. Thanks in advance.
------------------------------
Sasireka Dinesh
Original Message:
Sent: Thu January 25, 2024 11:10 AM
From: Eduardo Izquierdo Lázaro
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
Thanks for the answer Ross.
We know how to configure and tune ODM XU, we have been trying different settings of pool size, but eventually, during short time periods we have extremely high response times from ODM and all the client apps start receiving timeouts. This why I was wondering if maybe some other configuration parameter in liberty may help.
In our case, we have 10-11 millions request per day for around 60 decision services deployed in ODM. There are 2 nodesx3cores balanced. Some services are complex and take around 150 msec average. But most of them are very fast (10-20 msec average).
But when the performance degrades, even a very simple service decision service with average response time of 1 msec start giving response times over 10 sec meassured in the acces.log of liberty (that is, accounting execution time in the XU + reading the request + wirting the request + whatever). However, in the RES console, the max execution time rarely goes over 100 msec in this period.
The max pool size is 250 in each node, we have a timeout of 1sg waiting for an engine in the pool, so the problem is not that request is waiting for the pool. Time should be in some other point. CPU average is ok, and has some peaks, but even with peaks is very rare for me passing from 1 msec up to 10 sec. It should be something more.
This is the reason why I was asking in this community
------------------------------
Eduardo Izquierdo Lázaro
Automation Architect
DECIDE
Madrid
609893677
Original Message:
Sent: Thu January 25, 2024 09:01 AM
From: Ross MacDougall
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
For ODM on Liberty, XU Connection Pool sizing is done by adjusting the maxPoolSize in the ra.xml in the DecisionService.war file, not in Liberty itself. The default value is 20. I typically externalize the ra.xml by extracting the file from the war, and pre-pending a new directory onto the CLASSPATH in the jvm.options file. I can then modify the ra.xml without repacking the .war.
Step 1. Create a directory in the server home directory
cd /opt/IBM/wlp/usr/servers/myServer
mkdir ./res
Step 2. Add the following to the jvm.options
-Xbootclasspath/p:./res
Step 3. Extract the ra.xml from the DecisionService.war file and place it into the ./res
I have a directive in my server.xml to auto expand my .war files. This helps with performance
cp ./apps/expanded/DecisionService.war/WEB-INF/classes/ra.xml ./res/ra.xml
Step 4. Modify ra.xml and restart the server.
Modify the defaultConnectionManagerProperties stanza in the ra.xml
Change the maxpoolsize value to your desired value. As an example
<config-property>
<config-property-name>defaultConnectionManagerProperties</config-property-name>
<config-property-type>java.lang.String</config-property-type>
<config-property-value>
pool.maxSize=80,pool.waitTimeout=-1
</config-property-value>
</config-property>
Restart your server. Verify through your RES console - Server Information. Select Log Level = DEBUG, and dump the XU. You should see new values.
Our Environment
We do 2M ODM invocations per day on a 4 node (4 CPUs per node) WLP config behind a load balancer. 50 ms response times. 400K-600K JSON request payloads.
Other things to look at would be Document Builder Pool and Transformation Pool in the ra.xml. I'd also look at max heap size for ODM in WLP.
------------------------------
Ross MacDougall
Original Message:
Sent: Mon January 22, 2024 08:39 AM
From: Eduardo Izquierdo Lázaro
Subject: What are default values of maxPoolSize of ConnectionManager in liberty
We're running services executed by IBM ODM and the system has a very high throughput (up to 9k request per minute). This is a cluster with 2 nodes x 3 cores balanced by a previous LoadBalancer.
We're having performance issues and we're not sure if the problem is in the configuration of ODM or liberty. We're trying different tunning in ODM but we're not sure abouthe liberty tunning.
What is the default value maxPoolSize of liberty 20.0.0.9?
What would it be a recommended value for this throughput?
Note that some request have a very heavy JSON payload (up to more than 1M). Is there any other parameter we can tune related to the parsing of request payload?
------------------------------
Eduardo Izquierdo Lázaro
Automation Architect
DECIDE
Madrid
609893677
------------------------------