AAC Runtime JVM Garbage Collection

 

Hi Community

Here is a haunting topic for Halloween...

When the monitoring is activated on the AAC Runtime, one can extract various performance metrics from the underlying JVM runtime. For reference go to this great 2018 blog post  https://www.ibm.com/blogs/security-identity-access/monitoring-federation-advanced-access/.

First question I have: What is the unit for GcTime ? It appears to be in seconds, but it is not so clear to me.

With Initial and Max Heap Values of 2048, it is unusual to have GC Time Ranging from 2-6 seconds ?

Can that GC Time duration directly impact our AAC /token response times, or is the GC algorithm running more in async mode than in synchronous mode?

When the GC process runs, are all ACC Runtime Threads paused all at once ?

What is the by default garbage collect algorithm configured in AAC Runtime, and are there other algorithm available to make the GC process less impactful on OAuth endpoints response times ?

Thanks

------------------------------
Sylvain Gilbert
------------------------------

Hi Sylvain,

This is indeed very interesting topic and a great blog from Leo. 

You are right to assume GcTime to be in seconds here.

The AAC and Federation runtime profile by default uses gencon as gc policy which is highly optimized for transactional application. As a transactional application AAC mostly creates short lived objects . With the gencon policy, the Java heap is divided into two main areas, the nursery area, where new objects are created and the tenure area, where objects are moved if they have reached tenure age. GC Scavenge operation runs on nursery area which relatively small space . It can stop the world but these operations are very fast and handled by multiple GC workers which divide the workload across the cores available.

Long story short , we have the best GC policy suited for AAC.

It is advised to monitor the GC count and adjust heap size if we see too frequent GC cycles.

We have further enhanced the monitoring capabilities in recent versions of ISVA. Since ISVA 10.0.1 , we have added a "/metrics" endpoint which expose the JVM PerfMXBean counters in Prometheus friendly format for monitoring and graphing purpose. Details can be found at https://www.ibm.com/docs/es/sva/10.0.1?topic=settings-runtime-monitoring-using-prometheus.

Thanks,
Soum


------------------------------
Soumyajit Roy
------------------------------

Hi Soum

So we know the metric unit is CPU time in "second" but what about its scale ? Is it really seconds, milli-seconds ou micro-seconds ?

I'm starting to think that it cannot be 4-6 seconds of GC time on each run.

How can we be sure about it ?

------------------------------
Sylvain Gilbert
------------------------------

Original Message:
Sent: Mon October 31, 2022 06:34 AM
From: Soumyajit Roy
Subject: ISVA AAC Runtime JVM Garbage Collection

Hi Sylvain,

This is indeed very interesting topic and a great blog from Leo. 

You are right to assume GcTime to be in seconds here.

The AAC and Federation runtime profile by default uses gencon as gc policy which is highly optimized for transactional application. As a transactional application AAC mostly creates short lived objects . With the gencon policy, the Java heap is divided into two main areas, the nursery area, where new objects are created and the tenure area, where objects are moved if they have reached tenure age. GC Scavenge operation runs on nursery area which relatively small space . It can stop the world but these operations are very fast and handled by multiple GC workers which divide the workload across the cores available.

Long story short , we have the best GC policy suited for AAC.

It is advised to monitor the GC count and adjust heap size if we see too frequent GC cycles.

We have further enhanced the monitoring capabilities in recent versions of ISVA. Since ISVA 10.0.1 , we have added a "/metrics" endpoint which expose the JVM PerfMXBean counters in Prometheus friendly format for monitoring and graphing purpose. Details can be found at https://www.ibm.com/docs/es/sva/10.0.1?topic=settings-runtime-monitoring-using-prometheus.

Thanks,
Soum


------------------------------
Soumyajit Roy
------------------------------

Hi Sylvain,

The total time is across all the available CPU cores. So you need to divide it by the number of cores available to the system to get the real time. How many CPU cores are available in the system, you have captured the data from?

You can add a key -verbose:gc in jvm.options via advance tuning parameter to get more details on gc cycles. We should only care about stop the world pauses which suspends the application threads in terms of response time debugging.

Thanks,
Soum

------------------------------
Soumyajit Roy
------------------------------

Original Message:
Sent: Mon October 31, 2022 09:02 PM
From: Sylvain Gilbert
Subject: ISVA AAC Runtime JVM Garbage Collection

Hi Soum

So we know the metric unit is CPU time in "second" but what about its scale ? Is it really seconds, milli-seconds ou micro-seconds ?

I'm starting to think that it cannot be 4-6 seconds of GC time on each run.

How can we be sure about it ?

------------------------------
Sylvain Gilbert
------------------------------


Original Message:
Sent: Mon October 31, 2022 06:34 AM
From: Soumyajit Roy
Subject: ISVA AAC Runtime JVM Garbage Collection

Hi Sylvain,

This is indeed very interesting topic and a great blog from Leo. 

You are right to assume GcTime to be in seconds here.

The AAC and Federation runtime profile by default uses gencon as gc policy which is highly optimized for transactional application. As a transactional application AAC mostly creates short lived objects . With the gencon policy, the Java heap is divided into two main areas, the nursery area, where new objects are created and the tenure area, where objects are moved if they have reached tenure age. GC Scavenge operation runs on nursery area which relatively small space . It can stop the world but these operations are very fast and handled by multiple GC workers which divide the workload across the cores available.

Long story short , we have the best GC policy suited for AAC.

It is advised to monitor the GC count and adjust heap size if we see too frequent GC cycles.

We have further enhanced the monitoring capabilities in recent versions of ISVA. Since ISVA 10.0.1 , we have added a "/metrics" endpoint which expose the JVM PerfMXBean counters in Prometheus friendly format for monitoring and graphing purpose. Details can be found at https://www.ibm.com/docs/es/sva/10.0.1?topic=settings-runtime-monitoring-using-prometheus.

Thanks,
Soum


------------------------------
Soumyajit Roy