Hello guys

Need some help to understand this issue. We have 2 linux box with 4 IS in each of them & 1 IS per box which takes the max load (90%) is restarting by itself since we have upgraded to 9.8 a year back. The linux box free memory was around 300 MB on three occasions and around 2-3 GB on other occurrences when the IS restarted… suspecting it to be due to lack of memory at linux VM level, we are working with server team to explore increasing this but could there be other points to check?

Error in the wrapper log is the below. No other notable errors seen in server log or error log.

“A fatal error has been detected by the Java Runtime Environment”:
INFO | jvm 1 | 2017/11/04 15:09:11 | #
INFO | jvm 1 | 2017/11/04 15:09:11 | # SIGSEGV (0xb) at pc=0x00007f307503d3d5, pid=2087, tid=139845757560576
INFO | jvm 1 | 2017/11/04 15:09:11 | #
INFO | jvm 1 | 2017/11/04 15:09:11 | # JRE version: Java™ SE Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
INFO | jvm 1 | 2017/11/04 15:09:11 | # Java VM: Java HotSpot™ 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
INFO | jvm 1 | 2017/11/04 15:09:11 | # Problematic frame:
INFO | jvm 1 | 2017/11/04 15:09:11 | # V [libjvm.so+0x5c13d5] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x45
STATUS | wrapper | 2017/11/04 15:09:15 | JVM received a signal UNKNOWN (6).
STATUS | wrapper | 2017/11/04 15:09:15 | JVM process is gone.
ERROR | wrapper | 2017/11/04 15:09:15 | JVM exited unexpectedly.
STATUS | wrapper | 2017/11/04 15:09:20 | Launching a JVM…
INFO | jvm 2 | 2017/11/04 15:09:20 | Java HotSpot™ 64-Bit Server VM warning: ignoring option MaxPermSize=1280M; support was removed in 8.0
INFO | jvm 2 | 2017/11/04 15:09:21 | WrapperManager: Initializing…

Below are some environment parameters for reference -

1. wM details
   OS : Linux | CPU type : x86_64 | OS version : 6.9

IS version -9.8.0.0
updates - IS_9.8_Core_Fix13| IS_9.8_SPM_Fix1 | IS_9.8_SPM_Fix2

Java - 1.8.0_71 (52.0) (25.71-b15, mixed mode)

2. Memory details
   Linux memory - Total memory : 32 GB, results of top command below -
   box 1 -
   Mem: 32883700k total, 28295504k used, 4588196k free, 1259552k buffers
   Swap: 8388604k total, 470232k used, 7918372k free, 5300472k cached
   box 2 -
   Mem: 32883700k total, 29890392k used, 2993308k free, 1718380k buffers
   Swap: 8388604k total, 991392k used, 7397212k free, 6309392k cached

3. JVM memory : the corresponding IS JVM used memory never crosses 75%, is monitored every 10min.

Any suggestions on what the issue could be? Let me know if you need additional details.

Attachments -
1.the free memory across all occurrences and min/max memory defined for each component per box for reference.
2.detailed error report file of the last occurrence

thanks,
Shreyus

analysis.xls (30.5 KB)
hs_err_pid2087.log (741 KB)