When using the compressed OOPS mode in the 64bit JVM environment (IBM Java 8 and later), How much native memory of JVM memory area could we use ? 

when -Xmx5g,  max usable native memory size ? 

when -Xmx3g, max usable native memory size ? 

Compressed references for IBM Java and IBM Semeru Runtimes limits the available native memory for some native backing of classes, threads and monitors: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=management-heap-allocation#compressed-references

So the major factor is how much virtual native memory is available below 4GB and the amount needed for some native backing of classes, threads and monitors. This depends on many factors such as your operating system, your -Xmx, etc. Depending on the size of -Xmx, the Java heap might be placed underneath 4GB virtual which will further constrain the space required for some of the native class, thread, and monitor backing.

However, for your question of "How much native memory of JVM memory area could we use" and "max usable native memory size", this does not apply to native memory allocations other than some of the native backing of classes, threads, and monitors by the JVM. For example, if your application calls JNI to call malloc, then that will usually go above 4GB virtual and therefore it is not constrained by the compressed references design, and then you are just simply constrained to general native memory limitations (e.g. RAM, ulimits, etc.). There are some exceptions such as if your application directly wants to allocate below 4GB virtual or on Windows there are some cases where Windows will prefer allocating lower in the address space which will then compete with the JVM class/thread/monitor allocations: https://learn.microsoft.com/en-us/windows/win32/memory/4-gigabyte-tuning

There are other considerations that are operating system specific.

So unfortunately there are no easy answers, but at a general level, you should not worry about total available native memory, but rather you should worry about ensuring enough space is available underneath 4GB virtual for some of the native backing required for your class, thread, and monitor demands of your application. You might need to set -Xgc:preferredHeapBase=0x100000000 if your -Xmx is small, and/or increase -Xmcrs to avoid other native libraries taking space underneath 4GB virtual, and/or reduce or limit the amount of classes, threads, and monitors your application allocates. See some other considerations here: https://publib.boulder.ibm.com/httpserv/cookbook/Troubleshooting_Recipes-Troubleshooting_OpenJ9_and_IBM_J9_Recipes-J9_Native_OutOfMemoryError_Recipe.html

When using the compressed OOPS mode in the 64bit JVM environment (IBM Java 8 and later), How much native memory of JVM memory area could we use ? 

when -Xmx5g,  max usable native memory size ? 

when -Xmx3g, max usable native memory size ? 

Thank you for your sincere reply.  I have another questions....Could you help me ? 

For example, in the case of hazelcast running on the JVM, it is known that cache data is stored in the native memory area. If only map information pointing to cache data exists in native heap memory, are there any restrictions?

And, No matter how large the Xmx size is, even if the Heap Base address for Java Heap memory is larger than 4G and smaller than 32G, can the naive heap memory be used only up to 4G ? otherwise, up to the Heap Base address ? 

Compressed references for IBM Java and IBM Semeru Runtimes limits the available native memory for some native backing of classes, threads and monitors: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=management-heap-allocation#compressed-references

So the major factor is how much virtual native memory is available below 4GB and the amount needed for some native backing of classes, threads and monitors. This depends on many factors such as your operating system, your -Xmx, etc. Depending on the size of -Xmx, the Java heap might be placed underneath 4GB virtual which will further constrain the space required for some of the native class, thread, and monitor backing.

However, for your question of "How much native memory of JVM memory area could we use" and "max usable native memory size", this does not apply to native memory allocations other than some of the native backing of classes, threads, and monitors by the JVM. For example, if your application calls JNI to call malloc, then that will usually go above 4GB virtual and therefore it is not constrained by the compressed references design, and then you are just simply constrained to general native memory limitations (e.g. RAM, ulimits, etc.). There are some exceptions such as if your application directly wants to allocate below 4GB virtual or on Windows there are some cases where Windows will prefer allocating lower in the address space which will then compete with the JVM class/thread/monitor allocations: https://learn.microsoft.com/en-us/windows/win32/memory/4-gigabyte-tuning

There are other considerations that are operating system specific.

So unfortunately there are no easy answers, but at a general level, you should not worry about total available native memory, but rather you should worry about ensuring enough space is available underneath 4GB virtual for some of the native backing required for your class, thread, and monitor demands of your application. You might need to set -Xgc:preferredHeapBase=0x100000000 if your -Xmx is small, and/or increase -Xmcrs to avoid other native libraries taking space underneath 4GB virtual, and/or reduce or limit the amount of classes, threads, and monitors your application allocates. See some other considerations here: https://publib.boulder.ibm.com/httpserv/cookbook/Troubleshooting_Recipes-Troubleshooting_OpenJ9_and_IBM_J9_Recipes-J9_Native_OutOfMemoryError_Recipe.html

When using the compressed OOPS mode in the 64bit JVM environment (IBM Java 8 and later), How much native memory of JVM memory area could we use ? 

when -Xmx5g,  max usable native memory size ? 

when -Xmx3g, max usable native memory size ?