WebSphere Application Server & Liberty

 View Only
  • 1.  What are default values of maxPoolSize of ConnectionManager in liberty

    Posted Mon January 22, 2024 08:39 AM

    We're running services executed by IBM ODM and the system has a very high throughput (up to 9k request per minute). This is a cluster with 2 nodes x 3 cores balanced by a previous LoadBalancer.

    We're having performance issues and we're not sure if the problem is in the configuration of ODM or liberty. We're trying different tunning in ODM but we're not sure abouthe liberty tunning.

    What is the default value maxPoolSize of liberty 20.0.0.9?

    What would it be a recommended value for this throughput?

    Note that some request have a very heavy JSON payload (up to more than 1M). Is there any other parameter we can tune related to the parsing of request payload?



    ------------------------------
    Eduardo Izquierdo Lázaro
    Automation Architect
    DECIDE
    Madrid
    609893677
    ------------------------------


  • 2.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted Thu January 25, 2024 06:08 AM

    maxPoolSize : coreThreads is 1:1

    ODM performance issues should be do must gather for diagnosing.



    ------------------------------
    LI MIN YU
    ------------------------------



  • 3.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted Thu January 25, 2024 09:02 AM
    Edited by Ross MacDougall Thu January 25, 2024 09:16 AM

    For ODM on Liberty, XU Connection Pool sizing is done by adjusting the maxPoolSize in the ra.xml in the DecisionService.war file, not in Liberty itself.  The default value is 20.  I typically externalize the ra.xml by extracting the file from the war, and pre-pending a new directory onto the CLASSPATH in the jvm.options file.  I can then modify the ra.xml without repacking the .war.

    Step 1.  Create a directory in the server home directory 

    cd /opt/IBM/wlp/usr/servers/myServer

    mkdir ./res

    Step 2. Add the following to the jvm.options

    -Xbootclasspath/p:./res

    Step 3.  Extract the ra.xml from the DecisionService.war file and place it into the ./res

    I have a directive in my server.xml to auto expand my .war files.  This helps with performance

    cp ./apps/expanded/DecisionService.war/WEB-INF/classes/ra.xml ./res/ra.xml

    Step 4. Modify ra.xml and restart the server.

    Modify the defaultConnectionManagerProperties stanza in the ra.xml 

    Change the maxpoolsize value to your desired value.  As an example

    <config-property>
                 <config-property-name>defaultConnectionManagerProperties</config-property-name>
                               <config-property-type>java.lang.String</config-property-type>
                                             <config-property-value>
                                                          pool.maxSize=80,pool.waitTimeout=-1
                                             </config-property-value>

    </config-property>

    Restart your server.  Verify through your RES console - Server Information.  Select Log Level = DEBUG, and dump the XU. You should see new values.

    Our Environment

    We do 2M ODM invocations per day on a 4 node (4 CPUs per node) WLP config behind a load balancer.  50 ms response times.  400K-600K JSON request payloads.  

    Other things to look at would be Document Builder Pool and Transformation Pool in the ra.xml.  I'd also look at max heap size for ODM in WLP.



    ------------------------------
    Ross MacDougall
    ------------------------------



  • 4.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted Thu January 25, 2024 11:11 AM

    Thanks for the answer Ross.

    We know how to configure and tune ODM XU, we have been trying different settings of pool size, but eventually, during short time periods we have extremely high response times from ODM and all the client apps start receiving timeouts. This why I was wondering if maybe some other configuration parameter in liberty may help.

    In our case, we have 10-11 millions request per day for around 60 decision services deployed in ODM. There are 2 nodesx3cores balanced. Some services are complex and take around 150 msec average. But most of them are very fast (10-20 msec average). 

    But when the performance degrades, even a very simple service decision service with average response time of 1 msec start giving response times over 10 sec meassured in the acces.log of liberty (that is, accounting execution time in the XU + reading the request + wirting the request + whatever). However, in the RES console, the max execution time rarely goes over 100 msec in this period. 

    The max pool size is 250 in each node, we have a timeout of 1sg waiting for an engine in the pool, so the problem is not that request is waiting for the pool. Time should be in some other point. CPU average is ok, and has some peaks, but even with peaks is very rare for me passing from 1 msec up to 10 sec. It should be something more.

    This is the reason why I was asking in this community



    ------------------------------
    Eduardo Izquierdo Lázaro
    Automation Architect
    DECIDE
    Madrid
    609893677
    ------------------------------



  • 5.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted 21 days ago

    Hello @Eduardo,

    Just following up on your question.

    Were you able to diagnose the problem and identify the root cause for your performance degradation ?

    We are facing a similar situation to handle around 400 req (Soap req) in a second. 

    Initially, we noticed the following error from XU, which made us bump the pool size to 180 (defaultConnectionManagerProperties, document builder pool and transformation pool). We, no longer see this XU error but the response times are higher.  

    ilog.rules.res.session.IlrSessionException: An error occurred while the rule session was called: com.ibm.rules.res.xu.internal.XUException: XU client error. com.ibm.rules.res.xu.spi.internal.LocalizedResourceAllocationException: The default connection manager pool is full. ilog.rules.res.xu.pool.IlrPoolException: The pool is full.

    On ODM810 nodes (WAS), for the same volume of payload, the average response time is 460 msec but on ODM812 (wlp) its about 10sec.

    We have 3 nodes * 4 cores balanced by an ALB.

    Any information, that you can share will be helpful. Thanks in advance.



    ------------------------------
    Sasireka Dinesh
    ------------------------------



  • 6.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted 20 days ago

    In Liberty, in general, if you are receiving an influx of requests instantly before there is a sufficient ramp up time then you might want to set the coreThreads and maxThreads on the default Executor.  If you are receiving 400 requests per second, maybe you want to set the coreThreads to something higher than 400 and the maxThreads to whatever your peak load can be.  More information: https://openliberty.io/docs/latest/thread-pool-tuning.html

    In addition, you may want to open a case with IBM to help find where the performance issue lies if this a ODM issue or Liberty issue.

    Brian



    ------------------------------
    Brian S Paskin
    Sr. Technology Engineer
    IBM Cloud Engineering
    ------------------------------



  • 7.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted 20 days ago

    Hi Sasireka, we think that the problem may be related to the limits at the OS level (if you are in Linux). Ross pointed us in this direction and we discover that servers provisioned by the customer have very low limits. You can check it by running the commands:

    ulimit -Sa (for the soft limits)

    ulimit -Ha (for the hard limits)

    We opened a support case to get some recommended settings and we got this:

    ------------------------------------------------------

    For Linux, we have below technote for recommended limits.

    Insufficient ulimit -u (NPROC) Value Causes Native OutOfMemory warning message to be issued

    https://www.ibm.com/support/pages/insufficient-ulimit-u-nproc-value-causes-native-outofmemory-warning-message-be-issued

    User Limits (in bytes except for NOFILE and NPROC)

    type soft limit hard limit

    RLIMIT_AS unlimited unlimited

    RLIMIT_CORE unlimited unlimited

    RLIMIT_CPU unlimited unlimited

    RLIMIT_DATA unlimited unlimited

    RLIMIT_FSIZE unlimited unlimited

    RLIMIT_LOCKS unlimited unlimited

    RLIMIT_MEMLOCK 65536 65536

    RLIMIT_NOFILE 65536 65536

    RLIMIT_NPROC 131072 131072

    -----------------------------------------------------------

    We don't know yet if this is a solution, because these settings can only be modified by root user, so we asked client to do it, but the bureaucracy is taking long and now we're in freeze period. 

    If you read the link sent by Brian, it warns about this problem. Normally liberty should manage properly the peaks by increasing/decreasing the threads, but if limited by OS, it can't.

    Good luck  



    ------------------------------
    Eduardo Izquierdo Lázaro
    Automation Architect
    DECIDE
    Madrid
    609893677
    ------------------------------



  • 8.  RE: What are default values of maxPoolSize of ConnectionManager in liberty

    Posted 4 days ago

    If you get thread dumps from the ODM JVMs during the period when responses are very slow, you can check whether threads are waiting for some resource before proceeding.

    You can also check whether requests are queuing at the Liberty thread pool waiting for a thread by running with thread pool trace active. The trace spec for that, to be added to Liberty server.xml, is:

      <logging consoleFormat="simple" traceSpecification="com.ibm.ws.threading.internal.ThreadPoolController=all=enabled" maxFileSize="1024" maxFiles="10" />

                          note: the maxFileSize and maxFiles should be adjusted to suit the available filesystem space


    If you get thread dumps and/or thread pool trace and would like help analyzing them, feel free to open a Liberty support case, directed to WAS SDK team - they are very familiar with analysis of Liberty/Java performance issues.

    ------------------------------------------------

    Gary DeVal

    WebSphere/Java performance engineering

    ------------------------------------------------



    ------------------------------
    Gary DeVal
    ------------------------------