IDS Workgroup
11.50.UC9GE
65GB mem on box
Two engines, one 27GB, one 20GB

I believe we're running into mem issues (too many segs, etc.) which is causing the box or engine to crash. Not sure which just yet. 

Situation: 

The prod engine crashed due to ADM VP issue: 

06/19/23 21:02:59  Fatal error in ADM VP at mt.c:14033

06/19/23 21:02:59  Unexpected virtual processor termination, pid = 216374, exit = 0xb

06/19/23 21:02:59  PANIC: Attempting to bring system down

The test engine died with no message, so I assume the box fell over. 

Q:

1. there are no assertion failures files. Default location is $INFORMIXDIR/tmp - can't remember if that gets cleared on a box bounce (didn't think so). 
2. attempting to cap the test engine SHMTOTAL from 0 (of course). No matter what value I use for SHMTOTAL, the engine won't come up and msg log says "14:51:49  Value of SHMTOTAL exceeds operating system limits.". I used very small (which I expect to die since buffers is larger than what I used), and large as well, but same result. Pertinent kernel limits below:

------------------------------
Thanks! 
Mark Scranton
The Mark Scranton Group
mark@markscranton.com
------------------------------

I'd say "exit = 0xb" indicates this (adm) vp received a signal 11, i.e. SegV.  Not clear how this would be related to (too) many shmem segments...  is there an assertion and an af file?

------------------------------
Andreas Legner
------------------------------

Thanks Andreas ...

No assertion failure files oddly enough. 

Not sure there were "too many segments" - just a behavior I've seen with this engine. Looking at data though it appears there wasn't a huge memory allocation on the box at the time.

Mostly just stumped about Q2 in my post and the inability to cap mem with SHMTOTAL. Not suggesting this is the cause for this crash - but I need to be able to cap these engines w/ SHMTOTAL as I've seen as many as 91 segments allocated for their prod engine. I'll be changing SHMVIRTSIZE as soon as we bounce again of course to reduce the newly allocated segment number. 

Thanks!
Mark
mark@markscranton.com 

------------------------------
Thanks! 
Mark Scranton
The Mark Scranton Group
mark@markscranton.com
------------------------------

Original Message:
Sent: Thu June 22, 2023 10:00 AM
From: Andreas Legner
Subject: Linux Operating System Limits/AF files missing

I'd say "exit = 0xb" indicates this (adm) vp received a signal 11, i.e. SegV.  Not clear how this would be related to (too) many shmem segments...  is there an assertion and an af file?

------------------------------
Andreas Legner
------------------------------

Hi Mark,

I can't answer the SHMTOTAL question, but it looks to me like the ADM VP may have been killed by the Linux out-of-memory killer.  We see that fairly often when the engine dies with no associated assert failure.  You can run "dmesg" to see if there were any incidents of a process being killed because the server was "Out of memory".

Thanks,

Barrie

------------------------------
Barrie Shaw
------------------------------

That I'd assume would result in "exit = 0x9", i.e. the process being sent a signal 9.

------------------------------
Andreas Legner
------------------------------

Original Message:
Sent: Thu June 22, 2023 10:09 AM
From: Barrie Shaw
Subject: Linux Operating System Limits/AF files missing

Hi Mark,

I can't answer the SHMTOTAL question, but it looks to me like the ADM VP may have been killed by the Linux out-of-memory killer.  We see that fairly often when the engine dies with no associated assert failure.  You can run "dmesg" to see if there were any incidents of a process being killed because the server was "Out of memory".

Thanks,

Barrie

------------------------------
Barrie Shaw
------------------------------

Good point.  I stand corrected.

Thanks,

Barrie

------------------------------
Barrie Shaw
------------------------------

Original Message:
Sent: Thu June 22, 2023 12:31 PM
From: Andreas Legner
Subject: Linux Operating System Limits/AF files missing

That I'd assume would result in "exit = 0x9", i.e. the process being sent a signal 9.

------------------------------
Andreas Legner
------------------------------


Original Message:
Sent: Thu June 22, 2023 10:09 AM
From: Barrie Shaw
Subject: Linux Operating System Limits/AF files missing

Hi Mark,

I can't answer the SHMTOTAL question, but it looks to me like the ADM VP may have been killed by the Linux out-of-memory killer.  We see that fairly often when the engine dies with no associated assert failure.  You can run "dmesg" to see if there were any incidents of a process being killed because the server was "Out of memory".

Thanks,

Barrie

------------------------------
Barrie Shaw