Informix

Informix

Connect with Db2, Informix, Netezza, open source, and other data experts to gain value from your data, share insights, and solve problems.

 View Only
Back to discussions
Expand all | Collapse all

Memory block header in pool 'rsam' corruption

  • 1.  Memory block header in pool 'rsam' corruption

    Posted Tue July 09, 2024 04:11 AM
    Edited by Dennis Melnikov Tue July 09, 2024 06:26 AM

    Hi,

    IBM Informix Dynamic Server Version 11.70.FC5XE

    We got 5 occasions of memory block header corruption in less than 2 hours, all in pool 'rsam'.

    Two of these occasions generated AFs that emerged in online.log. And these 2 occasions appeared simultaneously with dbs_partn mutex contention with more than 100 sessions in wait list.

    What is it, what did they resulted from?

    11:07:35  Found during mt_shm_free 1
    11:07:35  Pool 'rsam' (0x700001f0fe3a040)
    11:07:35  Bad block header 0x700003f19602d50
    blk-64
    0700003f19602d10: 0e00007e 32c05a51 0010197c 00000e0e   ...~2.ZQ ...|....
    0700003f19602d20: 0700003f 19602d50 0700003f 19602cc0   ...?.`-P ...?.`,.
    0700003f19602d30: 00000000 00000040 00000000 0000ba51   .......@ .......Q
    0700003f19602d40: 00000150 00000008 00000000 00000000   ...P.... ........
    blk+64
    0700003f19602d50: 0e00007e 32c05a61 0014046c 00000df5   ...~2.Za ...l....
    0700003f19602d60: 0700003f 19602dd0 0700003f 19602d10   ...?.`-. ...?.`-.
    0700003f19602d70: 00000000 00000080 01040219 00010004   ........ ........
    0700003f19602d80: 00010000 01f03b0e 00000000 00000001   ......;. ........


    11:00:21  Found during mt_shm_free 1
    11:00:21  Pool 'rsam' (0x700001f0fe3a040)
    11:00:21  Bad block header 0x7000050d20c7fb8
    blk-64
    07000050d20c7f78: 0e0000a1 a418ff21 0010197c 00000e0e   .......! ...|....
    07000050d20c7f88: 07000050 d20c7fb8 07000050 d20c7f28   ...P.... ...P...(
    07000050d20c7f98: 00000000 00000040 00000000 00003e5f   .......@ ......>_
    07000050d20c7fa8: 00000078 00000008 00000000 00000000   ...x.... ........
    blk+64
    07000050d20c7fb8: 0e00009a 64c9dac1 0010197c 00000e0e   ....d... ...|....
    07000050d20c7fc8: 07000049 92cf6000 07000050 d20c7f78   ...I..`. ...P...x
    07000050d20c7fd8: 00000000 00000048 00000000 0002cd08   .......H ........
    07000050d20c7fe8: 00000008 00000008 00000000 00000000   ........ ........

    10:36:26  Found during mt_shm_free 1
    10:36:26  Pool 'rsam' (0x700001f0fe3a040)
    10:36:26  Bad block header 0x70000440a7dfea0
    blk-64
    070000440a7dfe60: 0e000088 14fbfcf1 0010197c 00000e0e   ........ ...|....
    070000440a7dfe70: 07000044 0a7dfea0 07000044 0a7dfe10   ...D.}.. ...D.}..
    070000440a7dfe80: 00000000 00000040 00000000 00000ea1   .......@ ........
    070000440a7dfe90: 00000005 00000008 00000000 00000000   ........ ........
    blk+64
    070000440a7dfea0: 0e000088 147bfda1 001019a6 00000e0e   .....{.. ........
    070000440a7dfeb0: 07000044 0a7dfef0 07000044 0a7dfe60   ...D.}.. ...D.}.`

    10:16:55  Found during mt_shm_free 1
    10:16:55  Pool 'rsam' (0x700001f0fe3a040)
    10:16:55  Bad block header 0x70000427e53ae50
    blk-64
    070000427e53ae10: 0e000084 fca75c41 00140e22 00000dc3   ......\A ..."....
    070000427e53ae20: 07000042 7e53ae50 07000042 7e53adb0   ...B~S.P ...B~S..
    070000427e53ae30: 00000000 00000040 0700003b 5e0019b8   .......@ ...;^...
    070000427e53ae40: 00000000 00000000 0e000054 cc5ae8b9   ........ ...T.Z..
    blk+64
    070000427e53ae50: 0e000084 fc275d21 00140e50 00000dc3   .....']! ...P....
    070000427e53ae60: 07000042 7e53aeb0 07000042 7e53ae10   ...B~S.. ...B~S..
    070000427e53ae70: 00000000 00000060 00000000 00000000   .......` ........
    070000427e53ae80: 00000000 00000000 00000000 00000000   ........ ........
    10:16:55  Multiple block errors found

    09:59:35  Found during mt_shm_free 1
    09:59:35  Pool 'rsam' (0x700001f0fe3a040)
    09:59:35  Bad block header 0x7000047b4510740
    blk-64
    07000047b4510700: 0e00008f 68a20e31 0010197c 00000e0e   ....h..1 ...|....
    07000047b4510710: 07000047 b4510740 07000047 b45106b0   ...G.Q.@ ...G.Q..
    07000047b4510720: 00000000 00000040 00000000 00015d92   .......@ ......].
    07000047b4510730: 00000005 00000008 00000000 00000000   ........ ........
    blk+64
    07000047b4510740: 0e00008f 68a20f51 0014046c 00000df5   ....h..Q ...l....
    07000047b4510750: 03000047 b45107c8 07000047 b4510700   ...G.Q.. ...G.Q..
    07000047b4510760: 00000000 00000088 01040218 00010004   ........ ........
    07000047b4510770: 00010000 008040d0 00000000 00000001   ......@. ........
    09:59:35  Bad block removed from pool
    09:59:35  Attempting to clean up free list...



    ------------------------------
    Sincerely,
    Dennis
    ------------------------------



  • 2.  RE: Memory block header in pool 'rsam' corruption

    Posted Tue July 09, 2024 07:27 AM

    Dennis:

    You are on a very old and very early release of v11.70 (FC5). It is highly likely that this was fixed in a later v11.70 release, but certainly is not an issue in 12.10 or 14.10. You need to upgrade!



    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------



  • 3.  RE: Memory block header in pool 'rsam' corruption

    Posted Wed July 10, 2024 04:43 AM

    If upgrade is not an option, what might be a workaround?

    We have plenty of rsam headers corrupted today.



    ------------------------------
    Sincerely,
    Dennis
    ------------------------------

    Original Message


  • 4.  RE: Memory block header in pool 'rsam' corruption

    Posted Wed July 10, 2024 05:03 AM

    Other than getting on a version that has the fix in it, your only recourse is to bounce the instance and hope it doesn't happen again. 

    Art



    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 5.  RE: Memory block header in pool 'rsam' corruption

    Posted Wed July 10, 2024 10:10 AM
    Edited by Dennis Melnikov Wed July 10, 2024 10:17 AM

    BTW, what does RSAM mean? The docs don't give any help.

    Does it stand for Random Sequential Access Method?

    ------------------------------
    Sincerely,
    Dennis
    ------------------------------

    Original Message


  • 6.  RE: Memory block header in pool 'rsam' corruption

    Posted Wed July 10, 2024 10:49 AM

    Denis:

    OK, so the original database sold be "Relational Systems Inc." which later became Informix, was an CISAM (Indexed Sequential Access Method for "C") library API for creating ISAM database tables. It was used to create the original Informix database system (known for years as Informix Standard Engine) which shipped free included when you bought 4GL or the ISAM development system. They later broke the engine out as a separate product. When Informix Corp. developed Informix Turbo, the first shared memory RDBMS, the CISAM library was not sufficient so they created a new IO library to use and dubbed it RSAM for Relational Sequential Access Method. That, or a vastly improved and expanded version of it, is still the underlying IO library underneath the Informix database system.

    Art



    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 7.  RE: Memory block header in pool 'rsam' corruption

    Posted Thu October 24, 2024 02:46 AM
    Edited by Dennis Melnikov Thu October 24, 2024 03:38 AM

    Hi,

    I feel like I must inform you that we managed to discover that RSAM buffer header corruption results from UPDATE STATISTICS.

    The memory got corrupted and later a random session encounters the failure.

    Meanwhile, UPDATE STATISTICS FORCE does not corrupt memory.



    ------------------------------
    Sincerely,
    Dennis
    ------------------------------



  • 8.  RE: Memory block header in pool 'rsam' corruption

    Posted Fri October 25, 2024 05:26 AM

    Hi Dennis,

    I don't believe there is any difference between UPDATE STATISTICS with or without the FORCE keyword except that if FORCE is not specified the engine may decide that updating statistics is not necessary (and do nothing) based on the AUTO_STAT_MODE and STATCHANGE parameters.

    This looks a lot like:

    IT09609: RSAM MEMORY POOL CORRUPTION RUNNING UPDATE STATISTICS ON TABLE WITH USTLOW_SAMPLE ENABLED (SET TO 1)

    https://www.ibm.com/support/pages/apar/IT09609

    Are you using sampling for UPDATE  STATISTICS [LOW]?

    Ben.



    ------------------------------
    Benjamin Thompson
    ------------------------------

    Original Message


  • 9.  RE: Memory block header in pool 'rsam' corruption

    Posted Fri October 25, 2024 06:36 AM

    Ben,

    Thank you for the link.

    1. We don't use LOW.
    2. USTLOW_SAMPLE is set to 0 (zero).


    ------------------------------
    Sincerely,
    Dennis
    ------------------------------

    Original Message


  • 10.  RE: Memory block header in pool 'rsam' corruption

    Posted Fri October 25, 2024 07:04 AM

    Dennis,

    LOW is the default mode if you just run 'UPDATE STATISTICS'. Regardless, if USTLOW_SAMPLE is not set it cannot be that bug.

    Another possible candidate: IT03769: FAST CACHE PROBLEM WHEN DROPPING TABLE OR DATABASE CAN ASSERT CRASH WITH BAD BLOCK HEADER MT_SHM_FREE 1

    Does the stack trace match what you have, even in part?

    I am not sure I can help much more. If I remember correctly 11.70.FC5 is from around 2005. When did you drop support? Do you have any entitlement to the latest 7.31 release which I think was 11.70.UD10 or something like that?

    Ben.



    ------------------------------
    Benjamin Thompson
    ------------------------------

    Original Message


  • 11.  RE: Memory block header in pool 'rsam' corruption

    Posted Wed February 25, 2026 03:30 AM

    Benjamin,

    > Does the stack trace match what you have, even in part?

    Yes, it does.

    We  are still experiencing occasional RSAM corruptions that are back again after some time of absence.

    Does FAST CACHE have any settings? If so, cannot find such in the docs.

    I have an idea to find out a culprit, please let me know if it makes any sense.

    Taking into account that a culprit corrupts memory, perhaps, hours before a victim encounters it, we will dump buffer headers at regular basis to find out an owner of the corrupted area, later.

    If this idea makes sense, what should we dump, `onstat -g mem`,.. ? How do we get a memory block owner?



    ------------------------------
    Sincerely,
    Dennis
    ------------------------------

    Original Message


  • 12.  RE: Memory block header in pool 'rsam' corruption

    Posted Mon October 28, 2024 02:43 PM

    Parece que estás enfrentando problemas serios de corrupción de memoria en IBM Informix. La corrupción del encabezado del bloque de memoria, especialmente en el pool 'rsam', puede indicar problemas con la gestión o asignación de memoria. La contención simultánea de mutex sugiere un cuello de botella en el acceso a recursos compartidos, lo que podría estar contribuyendo a los fallos.

    Para solucionar el problema, considera revisar los registros de tu aplicación en busca de patrones que conduzcan a estos eventos. También podría ser útil revisar la configuración y los parámetros de asignación de memoria. Si el problema persiste, contactar al soporte de IBM para obtener asistencia más específica podría ser beneficioso.

    Por cierto, para la resolución de problemas de forma remota con tu equipo, 

    @AnyDesk Descarga Gratuita De Grieta 8.1.0  puede ser una gran ayuda. La versión gratuita permite un acceso rápido y colaboración mientras diagnosticas estos problemas. ¡Espero que lo resuelvas pronto!



    ------------------------------
    adam clark
    ------------------------------



Related Content