Hi, 

I've noticed sometimes a thread "POST /api/v1/ExecuteMDX" will hang and cause a server crash if the thread doesn't get canelled. Anyone has any idea what is causing this and how can we debug? 

The tm1 server machine has 80GB of RAM and our model normally consumes around 15-20GB.

This is from the logs before a crash:

1392 [14e] DEBUG 2022-07-19 06:11:46.652 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '1392' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(40)'.
1392 [14e] DEBUG 2022-07-19 06:11:57.319 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '1392' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(40)'.
2524 [145] DEBUG 2022-07-19 06:22:22.102 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '2524' of user 'USER2' executing request 'DELETE /api/v1/MemberSets(2)'.
448 [15f] DEBUG 2022-07-19 06:26:44.828 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '448' of user 'USER3' executing request 'POST /api/v1/ExecuteMDX'.
448 [15f] DEBUG 2022-07-19 06:26:46.107 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '448' of user 'USER3' executing request 'POST /api/v1/ExecuteMDX'.
2312 [145] DEBUG 2022-07-19 06:26:57.757 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '2312' of user 'USER2' executing request 'DELETE /api/v1/MemberSets(7)'.
2312 [c7] DEBUG 2022-07-19 06:28:03.965 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '2312' of user 'USER4' executing request 'DELETE /api/v1/Cellsets('aDoylfcWAIBaAAAg')'.
3696 [132] ERROR 2022-07-19 06:28:56.830 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193710 Mb, Total Garbage Memory = 86087 Mb
3696 [132] WARN 2022-07-19 06:28:56.835 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
3696 [132] DEBUG 2022-07-19 06:28:57.204 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '3696' of user 'USER5' executing request 'DELETE /api/v1/Cellsets('aDoyleoXAICZAAAg')'.
6052 [14e] DEBUG 2022-07-19 06:29:37.461 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '6052' of user 'USERA1' executing request 'DELETE /api/v1/_async('DHUG5W50B22NF4BwFMen-t7EfdZxN')'.
3056 [14e] DEBUG 2022-07-19 06:30:58.718 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '3056' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(54)'.
3056 [14e] DEBUG 2022-07-19 06:30:59.965 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '3056' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(54)'.
140 [15f] DEBUG 2022-07-19 06:34:37.577 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '140' of user 'USER3' executing request 'GET /api/v1/Dimensions('Cost Center')/Hierarchies('Cost Center')/Subsets('HK RET Direct Opex CC')'.
6756 [] ERROR 2022-07-19 06:43:39.735 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
6756 [] WARN 2022-07-19 06:43:39.752 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7780 [] ERROR 2022-07-19 06:43:40.627 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
7780 [] WARN 2022-07-19 06:43:40.645 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
6328 [] ERROR 2022-07-19 06:43:41.078 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
6328 [] WARN 2022-07-19 06:43:41.425 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7032 [] ERROR 2022-07-19 06:43:41.820 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
7032 [] WARN 2022-07-19 06:43:41.926 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7144 [] ERROR 2022-07-19 06:43:42.057 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193729 Mb, Total Garbage Memory = 88350 Mb
7996 [145] ERROR 2022-07-19 06:43:42.057 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193729 Mb, Total Garbage Memory = 88350 Mb
7144 [] WARN 2022-07-19 06:43:42.065 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7996 [145] WARN 2022-07-19 06:43:42.065 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
4100 [] ERROR 2022-07-19 06:43:42.063 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193729 Mb, Total Garbage Memory = 88350 Mb
4100 [] WARN 2022-07-19 06:43:42.066 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
4100 [] WARN 2022-07-19 06:43:42.071 TM1.Server.Network net_Create Low Memory
7996 [145] DEBUG 2022-07-19 06:43:42.568 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '7996' of user 'USER2' executing request 'POST /api/v1/ExecuteMDX'.
4100 [145] ERROR 2022-07-19 06:43:42.967 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193730 Mb, Total Garbage Memory = 88350 Mb
4100 [145] WARN 2022-07-19 06:43:42.972 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
4100 [145] DEBUG 2022-07-19 06:43:43.284 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '4100' of user 'USER2' executing request 'POST /api/v1/ExecuteMDX'.
2022-07-19 06:47:41 GMT Shared memory reader PID 7512 started, for shmem "tm1s.exe-6400_1"

------------------------------
Luiz gustavo Ribeiro
------------------------------

#PlanningAnalyticswithWatson

Hi Luiz,

I recommend that you open a case with IBM to help you investigate the crash. The support team may ask you to create a dump file and send it to them. 
They usually review the dump file and tell you what event led to the crash.  

Link to IBM Support: IBM Support

Regards,
Kong

------------------------------
Kong Intrasinghathong
Master Consultant
QueBIT Consulting

Analyze. Plan. Achieve.
https://quebit.com
------------------------------

Original Message:
Sent: Tue July 19, 2022 06:34 PM
From: Luiz gustavo Ribeiro
Subject: Out of Memory error - system crash

Hi, 

I've noticed sometimes a thread "POST /api/v1/ExecuteMDX" will hang and cause a server crash if the thread doesn't get canelled. Anyone has any idea what is causing this and how can we debug? 

The tm1 server machine has 80GB of RAM and our model normally consumes around 15-20GB.

This is from the logs before a crash:

1392 [14e] DEBUG 2022-07-19 06:11:46.652 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '1392' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(40)'.
1392 [14e] DEBUG 2022-07-19 06:11:57.319 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '1392' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(40)'.
2524 [145] DEBUG 2022-07-19 06:22:22.102 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '2524' of user 'USER2' executing request 'DELETE /api/v1/MemberSets(2)'.
448 [15f] DEBUG 2022-07-19 06:26:44.828 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '448' of user 'USER3' executing request 'POST /api/v1/ExecuteMDX'.
448 [15f] DEBUG 2022-07-19 06:26:46.107 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '448' of user 'USER3' executing request 'POST /api/v1/ExecuteMDX'.
2312 [145] DEBUG 2022-07-19 06:26:57.757 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '2312' of user 'USER2' executing request 'DELETE /api/v1/MemberSets(7)'.
2312 [c7] DEBUG 2022-07-19 06:28:03.965 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '2312' of user 'USER4' executing request 'DELETE /api/v1/Cellsets('aDoylfcWAIBaAAAg')'.
3696 [132] ERROR 2022-07-19 06:28:56.830 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193710 Mb, Total Garbage Memory = 86087 Mb
3696 [132] WARN 2022-07-19 06:28:56.835 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
3696 [132] DEBUG 2022-07-19 06:28:57.204 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '3696' of user 'USER5' executing request 'DELETE /api/v1/Cellsets('aDoyleoXAICZAAAg')'.
6052 [14e] DEBUG 2022-07-19 06:29:37.461 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '6052' of user 'USERA1' executing request 'DELETE /api/v1/_async('DHUG5W50B22NF4BwFMen-t7EfdZxN')'.
3056 [14e] DEBUG 2022-07-19 06:30:58.718 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '3056' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(54)'.
3056 [14e] DEBUG 2022-07-19 06:30:59.965 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '3056' of user 'USERA1' executing request 'DELETE /api/v1/MemberSets(54)'.
140 [15f] DEBUG 2022-07-19 06:34:37.577 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '140' of user 'USER3' executing request 'GET /api/v1/Dimensions('Cost Center')/Hierarchies('Cost Center')/Subsets('HK RET Direct Opex CC')'.
6756 [] ERROR 2022-07-19 06:43:39.735 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
6756 [] WARN 2022-07-19 06:43:39.752 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7780 [] ERROR 2022-07-19 06:43:40.627 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
7780 [] WARN 2022-07-19 06:43:40.645 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
6328 [] ERROR 2022-07-19 06:43:41.078 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
6328 [] WARN 2022-07-19 06:43:41.425 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7032 [] ERROR 2022-07-19 06:43:41.820 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193728 Mb, Total Garbage Memory = 88351 Mb
7032 [] WARN 2022-07-19 06:43:41.926 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7144 [] ERROR 2022-07-19 06:43:42.057 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193729 Mb, Total Garbage Memory = 88350 Mb
7996 [145] ERROR 2022-07-19 06:43:42.057 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193729 Mb, Total Garbage Memory = 88350 Mb
7144 [] WARN 2022-07-19 06:43:42.065 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
7996 [145] WARN 2022-07-19 06:43:42.065 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
4100 [] ERROR 2022-07-19 06:43:42.063 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193729 Mb, Total Garbage Memory = 88350 Mb
4100 [] WARN 2022-07-19 06:43:42.066 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
4100 [] WARN 2022-07-19 06:43:42.071 TM1.Server.Network net_Create Low Memory
7996 [145] DEBUG 2022-07-19 06:43:42.568 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '7996' of user 'USER2' executing request 'POST /api/v1/ExecuteMDX'.
4100 [145] ERROR 2022-07-19 06:43:42.967 TM1.Server.Memory Unable to allocate block from OS: Total Memory Allocated = 193730 Mb, Total Garbage Memory = 88350 Mb
4100 [145] WARN 2022-07-19 06:43:42.972 TM1.Server.Memory AllocateBigBlock() outOfMemory Exception <<< MEMORY_FATAL_LEVEL >>> - apifunc# "0"
4100 [145] DEBUG 2022-07-19 06:43:43.284 TM1.Lock.Exception CommitActionLog::Rollback: Called for thread '4100' of user 'USER2' executing request 'POST /api/v1/ExecuteMDX'.
2022-07-19 06:47:41 GMT Shared memory reader PID 7512 started, for shmem "tm1s.exe-6400_1"

------------------------------
Luiz gustavo Ribeiro
------------------------------

#PlanningAnalyticswithWatson