Informix

 View Only

  • 1.  Memory issue on Informix primary server

    Posted Mon February 14, 2022 04:02 AM
      |   view attached

    Dear All,

     

     We have upgrade our Informix server from 11.7 to Version 14.10.FC6X3 on Sun OS 5.11.  After that We are facing multiple memory segment issue on Primary server, but everything is normal on Secondary server.

     

    Before Upgrade there was only on Virtual memory segment.

     

    I have checked all informix resource thru onstat -g osi, top , CPU, Memory all. mostly all looks same.

     

    I've attached a word file for some Informix command output for both Informix Primary and Secondary server.

    Can someone check and help me to solve the issue ?

     

    what could the best option apart from increasing the shared memory.

     

    Thanks

    Amit Patel



    ------------------------------
    AMIT PATEL
    ------------------------------

    #Informix

    Attachment(s)

    doc
    Informix_Resource.doc   86 KB 1 version


  • 2.  RE: Memory issue on Informix primary server

    IBM Champion
    Posted Mon February 14, 2022 06:38 AM
    Amit:

    All of those additional virtual segments are in use. I think that the only thing you can do is merge them all into the initial SHMVIRTSIZE and increase the SHMADD size by 10X or more. Solaris is REALLY slow to allocate shared memory, so the fewer segments the better.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 3.  RE: Memory issue on Informix primary server

    Posted Mon February 14, 2022 10:54 PM
    Thanks Art, 

           But total of virtual segment is 518617300992 Bytes means (518 GB) , which looks much to replace with current total shared memory.
     

    SHMVIRTSIZE 31457280 (31GB) and as per "onstat -g osi" system memory is System memory   781568 MB.

    So is there any other resource we can change Art?
    Or we can check to restart to production server to check if resource can be cleared.

    Thanks
    Amit Patel



    ------------------------------
    AMIT PATEL
    ------------------------------

    Original Message


  • 4.  RE: Memory issue on Informix primary server

    IBM Champion
    Posted Tue February 15, 2022 06:27 AM
    Amit:

    Casey's right, it looks like most of the memory allocated is unused. I had missed that.

    Here's what I think happened. As I said Solaris is VERY SLOW to allocate shared memory, so when a session needed more memory the engine requested another 3GB segment. It took so long that another session wanted more memory before the segment was completed which just slowed the process even more, and another session, and another, etc. until 500GB had been allocated. 

    What you can try is to increase SHMVIRTSIZE by 6 to 12 GB up from 32GB to say 44 GB and see how that goes. If it is not enough and the engine is still adding segments, then bump it again until that stops happening.

    The only way to stop the engine from creating new shared memory segments altogether would be to cap the maximum allocation with the SHMTOTAL parameter. But I don't think that is needed.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 5.  RE: Memory issue on Informix primary server

    Posted Mon February 14, 2022 11:41 PM
    Looking at the onstat -g seg output, currently those additonal virtual segments appears to be mostly unused. It is likely that there were sessions that were using a large amount of shared memory and triggered the engine to dynamically allocate additonal shared memory segments at one time. You can find out the timing on when these virtual segments were added from online.log, and try to find out if there are any special task/job running around the time.

    As the session/pool that was using the large amount memory has disconnected from the server, the onstat output would not show anything useful.  

    Running onmode -F might be able to free up virtual segments if there are no other resources using it, or restaring the engine will be able to release the additional virtual segments.

    Segment Summary:

    id         key        addr             size             ovhd     class blkused  blkfree
    6          52574801   10a000000        1140850688       13811032 R*    276082   2446
    7          52574802   14e000000        32212254720      377489352 V*    2153941  5710379
    8          52574803   8ce000000        6375342080       1        B*    1556480  0
    9          52574804   a4a000000        33554432         1        B*    8192     0
    10         52574805   a4c000000        3145728          38232    M     539      229
    11         52574806   a4c300000        3145728          38232    M     537      231
    12         52574807   a4d000000        3221225472       37750584 V*    10461    775971
    13         52574808   b0d000000        3221225472       37750584 V*    10153    776279
    14         52574809   bcd000000        3221225472       37750584 V*    9956     776476
    15         5257480a   c8d000000        3221225472       37750584 V*    9775     776657
    16         5257480b   d4d000000        3221225472       37750584 V*    9817     776615
    17         5257480c   e0d000000        3221225472       37750584 V*    10195    776237



    ------------------------------
    Casey Tan
    ------------------------------

    Original Message


  • 6.  RE: Memory issue on Informix primary server

    Posted Tue February 15, 2022 03:52 AM
    hi,
    with  14.10 you have more tasks running in the scheduler than with 11.70, may be you haven't launch it in version 11.70.
    Jacques


    ------------------------------
    jack rage
    ------------------------------

    Original Message


  • 7.  RE: Memory issue on Informix primary server

    Posted Sat February 26, 2022 08:44 AM
    I think there's a memory leak
    If you do not control, you will eventually run out of physical memory of the operating system
    I have encountered a similar case before
    informix 12.10fc4+redhat 6.5
    However, we found that it was caused by an application. When the application started to connect to the database, segment V began to add madly

    ------------------------------
    ZhiWei Cui
    GBASE
    ------------------------------

    Original Message


  • 8.  RE: Memory issue on Informix primary server

    IBM Champion
    Posted Sun February 27, 2022 06:23 AM

    Hi,

    You can either run "onstat -g mem" every few minutes from cron to hope to catch it.

    Or you can update the case statement in the alarmprogram so that when an extra shared memory segment is added it runs "onstat -g mem" to a file to you can see what is using the memory.

    You can also run "onstat -g ses" from cron every few minutes (10? 15?) to try to catch sessions which are using a lot of memory.

    If you find a session using a lot of memory then "onstat -g pqs" to see prepared statements and what the session is doing.

    David.

    ------------------------------
    David Williams
    ------------------------------

    Original Message


  • 9.  RE: Memory issue on Informix primary server

    Posted Mon February 28, 2022 10:37 AM
    I encourage anyone who needs a window into steady memory growth to check out onstat -g top, which is only available in 14.10:

    https://www.ibm.com/docs/en/informix-servers/14.10?topic=ogmo-onstat-g-top-command-print-top-consumers-resources

    General syntax:

    onstat -g top <keyword pair> <max consumers> <interval> <repeats>

    Example:

    onstat -g top mempool gro 10 30 0

    The above command will display the 10 memory pools that are growing the most, over 30 second intervals, repeating indefinitely.


    onstat -g top sessmem gro 5 20 3

    The above command will display the 5 sessions whose memory usage is growing the most, over 20 second intervals, repeating 3 times.


    This command will automatically sample, slice, and dice onstat data in all kinds of ways depending on which keyword pair you use.

    -jc

    ------------------------------
    John Lengyel
    ------------------------------

    Original Message