WebSphere Application Server & Liberty

 View Only
Expand all | Collapse all

Performance degradation on small user load

  • 1.  Performance degradation on small user load

    IBM Champion
    Posted Wed March 16, 2022 03:22 PM
    Hi,

    We are running Temenos core banking based on WebSphere 9.0.5.1 and MQ 9.1 with an Oracle Backend, we went to production and found that with heavy load the response time of the application is really high (in seconds). We went back to testing load on the platform, and we found out that with only 50 users we get the same behavior (knowing that we did some tuning Thread Pool, MQ Adapter, Activation Spec ...). 

    I started by analysing the heap (verboseGC), but everything seemed fine no issue there. When analysing the javacore, I've found multiple blocked thread (45/250 total threads), the thread are all WMQJCAResourceAdapter and all are blocked by the same thread which have the following stack : 

    State : Runnable
    Monitor : Owns Monitor Lock on com/ibm/ws/classloader/CompoundClassLoader@0x0000000800C963B0 
    Stack : 

    at java/io/UnixFileSystem.canonicalize0(Native Method)

    at java/io/UnixFileSystem.canonicalize(UnixFileSystem.java:183(Compiled Code))

    at java/io/File.getCanonicalPath(File.java:629(Compiled Code))

    at com/ibm/ws/classloader/SinglePathClassProvider.openZipFile(SinglePathClassProvider.java:174(Compiled Code))

    at com/ibm/ws/classloader/SinglePathClassProvider.openZipFile(SinglePathClassProvider.java:229(Compiled Code))

    at com/ibm/ws/classloader/SinglePathClassProvider.access$300(SinglePathClassProvider.java:72(Compiled Code))

    at com/ibm/ws/classloader/SinglePathClassProvider$2.run(SinglePathClassProvider.java:389(Compiled Code))

    at com/ibm/ws/security/util/AccessController.doPrivileged(AccessController.java:63(Compiled Code))

    at com/ibm/ws/classloader/SinglePathClassProvider.getClassBytesFromJar(SinglePathClassProvider.java:379(Compiled Code))

    at com/ibm/ws/classloader/SinglePathClassProvider.getClassBytes(SinglePathClassProvider.java:307(Compiled Code))

    at com/ibm/ws/classloader/CompoundClassLoader.localFindClass(CompoundClassLoader.java:735(Compiled Code))

    at com/ibm/ws/classloader/CompoundClassLoader.loadClass(CompoundClassLoader.java:622(Compiled Code))

    at java/lang/ClassLoader.loadClass(ClassLoader.java:865(Compiled Code)) 

    .... (removed as it contains confidential application stack)

    So my understanding is that it trying to load a class from the filesystem and it have issue related to this, cannot find another path to troubleshoot further.

    Any help would be appreciated.

    Thank you

    ----------------------
    Largou Walid
    Solution Architect / IBM Champion
    Power Maroc
    ----------------------





    ------------------------------
    Largou walid
    ------------------------------


  • 2.  RE: Performance degradation on small user load

    IBM Champion
    Posted Thu March 17, 2022 04:05 AM
    Hello Largou,

    Because is an java.io.UnixFileSystem any type of issue with storage IOs? a directory with a lot of files? something related to filesystem used at server level? how is the classpath and path ordered?

    Hope this helps.

    Regards

    ------------------------------
    Gabriel Aberasturi
    Versia tecnologias emergentes
    ------------------------------



  • 3.  RE: Performance degradation on small user load

    IBM Champion
    Posted Thu March 17, 2022 05:47 AM
    Hi,

    Thanks for your reply, indeed the filesystem contains 1 million file, but I don't understand why it's trying to load a class multiple times (45 times in 30s). I will try to change the class path load order and check if it resolves the issue.

    Thank you

    ------------------------------
    Largou walid
    ------------------------------



  • 4.  RE: Performance degradation on small user load

    Posted Thu March 17, 2022 04:08 AM
    Hi, the RA should have to only load the classes once if used properly.  Without seeing the stacks and trace it is hard to tell what exactly is going on.  However, there has been over 900+ fixes since the version of tWAS you are running.  I would suggest you try an upgrade and see if that fixes the issue, as a newer RA is included, update the RA manually, or open a case with IBM.

    Brian

    ------------------------------
    Brian S Paskin
    Sr. Web Engineer
    IBM Tech Garage
    ------------------------------



  • 5.  RE: Performance degradation on small user load

    IBM Champion
    Posted Thu March 17, 2022 05:49 AM
    Hi,

    Thanks for the reply, indeed we are planning to fix to latest version 9.0.5.10 in the coming days, but still want to confirm the issue. Can you suggest in trace activation that would give more insight for the class loading (we want to know which class it's trying to load).

    Thank you

    ------------------------------
    Largou walid
    ------------------------------



  • 6.  RE: Performance degradation on small user load

    Posted Thu March 17, 2022 06:08 AM
    Hi, you can follow this mustgather document for the tracing. I would also follow this mustgather for JMS/MQ issues.

    Brian

    ------------------------------
    Brian S Paskin
    Sr. Web Engineer
    IBM Tech Garage
    ------------------------------



  • 7.  RE: Performance degradation on small user load

    IBM Champion
    Posted Thu March 17, 2022 12:02 PM
    Hi,

    I already done the must gather on JMS and couldn't find something relevant, on the other side the class loading traces showed me in the trace this kind of messages (thousands of them for different jars) :

    [3/17/22 15:34:06:656 CET] 00000416 SinglePathCla 3 adding cached zip file <mypath>/myjar.jar
    [3/17/22 15:34:06:655 CET] 00000416 SinglePathCla 3 failed to read <my_class> from <mypath>/myjar.jar

    Is it normal behaviour ? 

    Thank you

    ------------------------------
    Largou walid
    ------------------------------



  • 8.  RE: Performance degradation on small user load

    Posted Thu March 17, 2022 12:10 PM
    Hi, that might be normal.  It is hard to say what the issue is for your snippet of the trace.  Where is this failing exactly?  During the JMS calls, or do you also have web services involved?  It might be a caching issue.

    Brian

    ------------------------------
    Brian S Paskin
    Sr. Web Engineer
    IBM Tech Garage
    ------------------------------



  • 9.  RE: Performance degradation on small user load

    Posted Thu March 17, 2022 12:14 PM
    If that specific class is actually in that specific jar, then it seems like something is misbehaving. If the class is not in that jar, but the jar does contain the same directory path, then the class loader will search the jar file to see if the .class file is present, and it will print that message to trace if it doesn't find it.

    For both that message and the "failed" message you mentioned in the other reply, those messages can often occur during normal class loading behavior, and you should only worry about them if you're seeing actual class loading failures. My understanding is that your issue here is not that class loads are failing, but rather that you're seeing lots of them, and they're causing a lot of activity on the file system.

    Were you able to identify what classes are being loaded at the time you're seeing the heavy file system activity, and if so, are you able to identify what code is requesting those class loads? I'm realizing as I type this that there is one possible reason to care about the "failed to read" message - if you've got a bunch of jars that all share a common set of packages, the application class loader will potentially search all of those jars for classes in the common packages. If this is where the bottleneck is occurring, you might want to consider repackaging your application such that classes in separate jars are also in separate packages, minimizing the number of jars that the class loader searches on each load.

    ------------------------------
    Jarid Kvale
    ------------------------------



  • 10.  RE: Performance degradation on small user load

    Posted Thu March 17, 2022 08:56 AM
    The classes/methods immediately below ClassLoader.loadClass() in the stack are also going to need to be a point of investigation eventually, as you will probably need to be inspect that code and see what exactly it's doing that could be causing class loading activity. The class loader trace Brian suggested is extremely useful, as it will tell you what classes are being loaded (search trace.log for "loadClass" around the time of the contention), but it won't tell you WHY they're being loaded, and that's ultimately going to be what you need to know in order to stop it from happening.

    ------------------------------
    Jarid Kvale
    ------------------------------



  • 11.  RE: Performance degradation on small user load

    IBM Champion
    Posted Thu March 17, 2022 12:05 PM
    Hi,

    I found multiple class failing with the message : 

    CompoundClass < loadClass <myclass> failed Exit

    I don't know what's the reason, the paths are correct.

    Thank you

    ------------------------------
    Largou walid
    ------------------------------