Dear All,

 

 We have upgrade our Informix server from 11.7 to Version 14.10.FC6X3 on Sun OS 5.11.  After that We are facing multiple memory segment issue on Primary server, but everything is normal on Secondary server.

 

Before Upgrade there was only on Virtual memory segment.

 

I have checked all informix resource thru onstat -g osi, top , CPU, Memory all. mostly all looks same.

 

I've attached a word file for some Informix command output for both Informix Primary and Secondary server.

Can someone check and help me to solve the issue ?

 

what could the best option apart from increasing the shared memory.

 

Thanks

Amit Patel

------------------------------
AMIT PATEL
------------------------------

#Informix

Amit:

All of those additional virtual segments are in use. I think that the only thing you can do is merge them all into the initial SHMVIRTSIZE and increase the SHMADD size by 10X or more. Solaris is REALLY slow to allocate shared memory, so the fewer segments the better.

Art

------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com
------------------------------

Thanks Art, 

       But total of virtual segment is 518617300992 Bytes means (518 GB) , which looks much to replace with current total shared memory.
 

SHMVIRTSIZE 31457280 (31GB) and as per "onstat -g osi" system memory is System memory   781568 MB.

So is there any other resource we can change Art?
Or we can check to restart to production server to check if resource can be cleared.

Thanks
Amit Patel

------------------------------
AMIT PATEL
------------------------------

Original Message:
Sent: Mon February 14, 2022 06:38 AM
From: Art Kagel
Subject: Memory issue on Informix primary server

Amit:

All of those additional virtual segments are in use. I think that the only thing you can do is merge them all into the initial SHMVIRTSIZE and increase the SHMADD size by 10X or more. Solaris is REALLY slow to allocate shared memory, so the fewer segments the better.

Art

------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com
------------------------------

Amit:

Casey's right, it looks like most of the memory allocated is unused. I had missed that.

Here's what I think happened. As I said Solaris is VERY SLOW to allocate shared memory, so when a session needed more memory the engine requested another 3GB segment. It took so long that another session wanted more memory before the segment was completed which just slowed the process even more, and another session, and another, etc. until 500GB had been allocated. 

What you can try is to increase SHMVIRTSIZE by 6 to 12 GB up from 32GB to say 44 GB and see how that goes. If it is not enough and the engine is still adding segments, then bump it again until that stops happening.

The only way to stop the engine from creating new shared memory segments altogether would be to cap the maximum allocation with the SHMTOTAL parameter. But I don't think that is needed.

Art

------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com
------------------------------

Original Message:
Sent: Mon February 14, 2022 10:54 PM
From: AMIT PATEL
Subject: Memory issue on Informix primary server

Thanks Art, 

       But total of virtual segment is 518617300992 Bytes means (518 GB) , which looks much to replace with current total shared memory.
 

SHMVIRTSIZE 31457280 (31GB) and as per "onstat -g osi" system memory is System memory   781568 MB.

So is there any other resource we can change Art?
Or we can check to restart to production server to check if resource can be cleared.

Thanks
Amit Patel

------------------------------
AMIT PATEL
------------------------------


Original Message:
Sent: Mon February 14, 2022 06:38 AM
From: Art Kagel
Subject: Memory issue on Informix primary server

Amit:

All of those additional virtual segments are in use. I think that the only thing you can do is merge them all into the initial SHMVIRTSIZE and increase the SHMADD size by 10X or more. Solaris is REALLY slow to allocate shared memory, so the fewer segments the better.

Art

------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com

Looking at the onstat -g seg output, currently those additonal virtual segments appears to be mostly unused. It is likely that there were sessions that were using a large amount of shared memory and triggered the engine to dynamically allocate additonal shared memory segments at one time. You can find out the timing on when these virtual segments were added from online.log, and try to find out if there are any special task/job running around the time.

As the session/pool that was using the large amount memory has disconnected from the server, the onstat output would not show anything useful.  

Running onmode -F might be able to free up virtual segments if there are no other resources using it, or restaring the engine will be able to release the additional virtual segments.

Segment Summary:

id         key        addr             size             ovhd     class blkused  blkfree
6          52574801   10a000000        1140850688       13811032 R*    276082   2446
7          52574802   14e000000        32212254720      377489352 V*    2153941  5710379
8          52574803   8ce000000        6375342080       1        B*    1556480  0
9          52574804   a4a000000        33554432         1        B*    8192     0
10         52574805   a4c000000        3145728          38232    M     539      229
11         52574806   a4c300000        3145728          38232    M     537      231
12         52574807   a4d000000        3221225472       37750584 V*    10461    775971
13         52574808   b0d000000        3221225472       37750584 V*    10153    776279
14         52574809   bcd000000        3221225472       37750584 V*    9956     776476
15         5257480a   c8d000000        3221225472       37750584 V*    9775     776657
16         5257480b   d4d000000        3221225472       37750584 V*    9817     776615
17         5257480c   e0d000000        3221225472       37750584 V*    10195    776237

------------------------------
Casey Tan
------------------------------

------------------------------
jack rage
------------------------------

I think there's a memory leak
If you do not control, you will eventually run out of physical memory of the operating system
I have encountered a similar case before
informix 12.10fc4+redhat 6.5
However, we found that it was caused by an application. When the application started to connect to the database, segment V began to add madly

------------------------------
ZhiWei Cui
GBASE
------------------------------

Hi,

You can either run "onstat -g mem" every few minutes from cron to hope to catch it.

Or you can update the case statement in the alarmprogram so that when an extra shared memory segment is added it runs "onstat -g mem" to a file to you can see what is using the memory.

You can also run "onstat -g ses" from cron every few minutes (10? 15?) to try to catch sessions which are using a lot of memory.

If you find a session using a lot of memory then "onstat -g pqs" to see prepared statements and what the session is doing.

David.

------------------------------
David Williams
------------------------------

I encourage anyone who needs a window into steady memory growth to check out onstat -g top, which is only available in 14.10:

https://www.ibm.com/docs/en/informix-servers/14.10?topic=ogmo-onstat-g-top-command-print-top-consumers-resources

General syntax:

onstat -g top <keyword pair> <max consumers> <interval> <repeats>

Example:

onstat -g top mempool gro 10 30 0

The above command will display the 10 memory pools that are growing the most, over 30 second intervals, repeating indefinitely.


onstat -g top sessmem gro 5 20 3

The above command will display the 5 sessions whose memory usage is growing the most, over 20 second intervals, repeating 3 times.


This command will automatically sample, slice, and dice onstat data in all kinds of ways depending on which keyword pair you use.

-jc

------------------------------
John Lengyel
------------------------------