Java, Semeru Runtimes and Runtimes for Business

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

Hi Steven,

This might be related to the Control Flow Guard (CFG) feature in Windows. We have seen cases where the OpenJ9 JIT compiler sometimes operates in a way that CFG doesn't like.

A fix is on the way for that issue which should be included in the next IBM Semeru release in Q2 2025, but in the meantime you could try the following workarounds:

• Use the -Xrs Java command line option
• Use the -Xjit:disableTraps Java command line option
• Disable CFG

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

Hello Paul, 

Thanks for the response.  I was testing this yesterday afternoon and it does appear to be the issue.  When I set -XJit:count=0 the issue goes from intermittent to crashing shortly after startup.  Using the -Xjit:disableTraps appears to fix it.  My issue now is given those options what are the repercussions of them?  I can't seem to find much documentation on what they are doing and what the consequences of using them might be.  We have a very large enterprise application so it would be nice to know what to look for in testing these options. We would like to refrain from disabling the Windows protection feature in CFG.

Thanks!

Hi Steven,

This might be related to the Control Flow Guard (CFG) feature in Windows. We have seen cases where the OpenJ9 JIT compiler sometimes operates in a way that CFG doesn't like.

A fix is on the way for that issue which should be included in the next IBM Semeru release in Q2 2025, but in the meantime you could try the following workarounds:

• Use the -Xrs Java command line option
• Use the -Xjit:disableTraps Java command line option
• Disable CFG

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

Hello Paul, 

Thanks for the response.  I was testing this yesterday afternoon and it does appear to be the issue.  When I set -XJit:count=0 the issue goes from intermittent to crashing shortly after startup.  Using the -Xjit:disableTraps appears to fix it.  My issue now is given those options what are the repercussions of them?  I can't seem to find much documentation on what they are doing and what the consequences of using them might be.  We have a very large enterprise application so it would be nice to know what to look for in testing these options. We would like to refrain from disabling the Windows protection feature in CFG.

Thanks!

Hi Steven,

This might be related to the Control Flow Guard (CFG) feature in Windows. We have seen cases where the OpenJ9 JIT compiler sometimes operates in a way that CFG doesn't like.

A fix is on the way for that issue which should be included in the next IBM Semeru release in Q2 2025, but in the meantime you could try the following workarounds:

• Use the -Xrs Java command line option
• Use the -Xjit:disableTraps Java command line option
• Disable CFG

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

Hi Steven,

This might be related to the Control Flow Guard (CFG) feature in Windows. We have seen cases where the OpenJ9 JIT compiler sometimes operates in a way that CFG doesn't like.

A fix is on the way for that issue which should be included in the next IBM Semeru release in Q2 2025, but in the meantime you could try the following workarounds:

• Use the -Xrs Java command line option
• Use the -Xjit:disableTraps Java command line option
• Disable CFG

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

How would that compare to the -Xrs option?  Is it indeed just a optimization or will we lose some form of error detection by disabling? 

I appreciate the information!

Steve

------------------------------
Steven Allen

Original Message:
Sent: Fri March 28, 2025 07:15 AM
From: Paul Cheeseman
Subject: Upgrading app from Java8 (Oracle) to IBM Semeru 17.0.14. Intermittent application crashes.

Hi Steven,

This might be related to the Control Flow Guard (CFG) feature in Windows. We have seen cases where the OpenJ9 JIT compiler sometimes operates in a way that CFG doesn't like.

A fix is on the way for that issue which should be included in the next IBM Semeru release in Q2 2025, but in the meantime you could try the following workarounds:

• Use the -Xrs Java command line option
• Use the -Xjit:disableTraps Java command line option
• Disable CFG

------------------------------
Paul Cheeseman

Original Message:
Sent: Thu March 27, 2025 09:58 AM
From: Steven Allen
Subject: Upgrading app from Java8 (Oracle) to IBM Semeru 17.0.14. Intermittent application crashes.

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

------------------------------
Steven Allen
------------------------------

-Xrs is much broader and less favourable if -Xjit:disableTraps is sufficient. -Xrs disables nearly all of the JVM's signal handling, which includes the traps used by the JIT, but much more besides (e.g. the JVM's signal handlers won't even be invoked if the JVM crashes).

-Xjit:disableTraps disables optimizations only. There will be no functional difference, just a small performance penalty when throwing certain Exceptions from JIT compiled code.

------------------------------
Paul Cheeseman (IBM Runtimes Support)

Original Message:
Sent: Fri March 28, 2025 01:02 PM
From: Steven Allen
Subject: Upgrading app from Java8 (Oracle) to IBM Semeru 17.0.14. Intermittent application crashes.

How would that compare to the -Xrs option?  Is it indeed just a optimization or will we lose some form of error detection by disabling? 

I appreciate the information!

Steve

------------------------------
Steven Allen

Original Message:
Sent: Fri March 28, 2025 07:15 AM
From: Paul Cheeseman
Subject: Upgrading app from Java8 (Oracle) to IBM Semeru 17.0.14. Intermittent application crashes.

Hi Steven,

This might be related to the Control Flow Guard (CFG) feature in Windows. We have seen cases where the OpenJ9 JIT compiler sometimes operates in a way that CFG doesn't like.

A fix is on the way for that issue which should be included in the next IBM Semeru release in Q2 2025, but in the meantime you could try the following workarounds:

• Use the -Xrs Java command line option
• Use the -Xjit:disableTraps Java command line option
• Disable CFG

------------------------------
Paul Cheeseman

Original Message:
Sent: Thu March 27, 2025 09:58 AM
From: Steven Allen
Subject: Upgrading app from Java8 (Oracle) to IBM Semeru 17.0.14. Intermittent application crashes.

Our team is currently upgrading our enterprise application from Java 8 to OpenJDK 17.0.14. Since the upgrade our users are experiencing seemingly random crashes producing an event log and crash.dmp file. No other errors logs are produced. The event in event viewer is as follows:

Faulting application name: java.exe, version: 17.0.14.0, time stamp: 0x050cc26d Faulting module name: ntdll.dll, version: 10.0.19041.5438, time stamp: 0xab0dece3 Exception code: 0xc0000409 Fault offset: 0x000000000006ee63 Faulting process id: 0x391c

Crash dump header reads:

ExceptionAddress: 00007ff9bd01ee63 (ntdll!RtlGuardRestoreContext+0x00000000000000b3)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 000000000000000d
Subcode: 0xd FAST_FAIL_INVALID_SET_OF_CONTEXT

We have users on both Windows 10 and Windows 11. Only Windows 10 users seem to be experiencing this crash.

Win10 Version: 10.0.19045 Build 19045

Further, the application does not crash on Java 8.

We have debugged with JNI scanning and memory analysis and found no errors within the application prior to crashing. Windows seems to be killing the process leaving Java unable to provide any error logs outside of stderror. The error produces no java exceptions or jni complaints.

Any tips to debugging this further?

I can provide the dump if requested.

------------------------------
Steven Allen
------------------------------