My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

Thank you very very much for the response first and it is also a very insightful one. 

Can you please let me know how i can find the threshold for topas and topas nmon mode. I just ran the command "topas" and i also thought since our application is not using much CPU it is not showing up. Then i did see some other application with a very low value of CPU but as you have mentioned our application might be consuming very low CPU as it was blocked. 

Can you please let me know how to take a snap? 

thanks,

ajit

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

Yes, i have access to the application code. 

The issue with restarting the application every hour is we dont know when the application is going to get stuck - so it could get stuck in the time gap between two restarts. One idea is to have a watch dog type thread that should keep checking for the process in topas and if this process is not in topas, do a restart. Please let me know what you think about this - may be a script can do this? This is not the ideal solution though. 

Thank you very very much for the response first and it is also a very insightful one. 

Can you please let me know how i can find the threshold for topas and topas nmon mode. I just ran the command "topas" and i also thought since our application is not using much CPU it is not showing up. Then i did see some other application with a very low value of CPU but as you have mentioned our application might be consuming very low CPU as it was blocked. 

Can you please let me know how to take a snap? 

thanks,

ajit

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

Thank you very very much for the response first and it is also a very insightful one. 

Can you please let me know how i can find the threshold for topas and topas nmon mode. I just ran the command "topas" and i also thought since our application is not using much CPU it is not showing up. Then i did see some other application with a very low value of CPU but as you have mentioned our application might be consuming very low CPU as it was blocked. 

Can you please let me know how to take a snap? 

thanks,

ajit

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

AIX Command Documentation https://www.ibm.com/docs/en/aix/7.3?topic=n-nmon-command

The -I option allows you to change the threshold and documents the default.

snap commanddocumentation is here https://www.ibm.com/docs/en/aix/7.3?topic=s-snap-command

snap -a

snap -c

It can take time. Be patient.

The pax file (which IBM AIX Support will want) ends up here /tmp/ibmsupt 

Make sure it has enough space in this file system.

At least a couple of GB. 

Then check the file system to see if it run out of space. 

If it has: double the size and redo.

Thank you very very much for the response first and it is also a very insightful one. 

Can you please let me know how i can find the threshold for topas and topas nmon mode. I just ran the command "topas" and i also thought since our application is not using much CPU it is not showing up. Then i did see some other application with a very low value of CPU but as you have mentioned our application might be consuming very low CPU as it was blocked. 

Can you please let me know how to take a snap? 

thanks,

ajit

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

Thank you. Will check on these. Once again, thank you for your time and very helpful suggestions. 

ajit

AIX Command Documentation https://www.ibm.com/docs/en/aix/7.3?topic=n-nmon-command

The -I option allows you to change the threshold and documents the default.

snap commanddocumentation is here https://www.ibm.com/docs/en/aix/7.3?topic=s-snap-command

snap -a

snap -c

It can take time. Be patient.

The pax file (which IBM AIX Support will want) ends up here /tmp/ibmsupt 

Make sure it has enough space in this file system.

At least a couple of GB. 

Then check the file system to see if it run out of space. 

If it has: double the size and redo.

Thank you very very much for the response first and it is also a very insightful one. 

Can you please let me know how i can find the threshold for topas and topas nmon mode. I just ran the command "topas" and i also thought since our application is not using much CPU it is not showing up. Then i did see some other application with a very low value of CPU but as you have mentioned our application might be consuming very low CPU as it was blocked. 

Can you please let me know how to take a snap? 

thanks,

ajit

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

One observation while debugging - the process started running and generating logs when we ran a command like ps - mo THREAD <pid> and/or a script to dump the thread stack trace. This did not happen always but it happened couple of times. 

Does the topas command agregrate the cpu utilization of all cores on the system?

thanks,

ajit

Thank you. Will check on these. Once again, thank you for your time and very helpful suggestions. 

ajit

AIX Command Documentation https://www.ibm.com/docs/en/aix/7.3?topic=n-nmon-command

The -I option allows you to change the threshold and documents the default.

snap commanddocumentation is here https://www.ibm.com/docs/en/aix/7.3?topic=s-snap-command

snap -a

snap -c

It can take time. Be patient.

The pax file (which IBM AIX Support will want) ends up here /tmp/ibmsupt 

Make sure it has enough space in this file system.

At least a couple of GB. 

Then check the file system to see if it run out of space. 

If it has: double the size and redo.

Thank you very very much for the response first and it is also a very insightful one. 

Can you please let me know how i can find the threshold for topas and topas nmon mode. I just ran the command "topas" and i also thought since our application is not using much CPU it is not showing up. Then i did see some other application with a very low value of CPU but as you have mentioned our application might be consuming very low CPU as it was blocked. 

Can you please let me know how to take a snap? 

thanks,

ajit

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call. 

Nigel,

BTW, what are pitfalls of running AIX on less than 16 GB RAM?

You do not state the details of your topas command.

topas and topas in nmon mode have a threshold so they do not report processes using zero CPU time (actually under the threshold of a tiny fraction of a CPU),
as saving millions of zeros is rather pointless in CPU cycles and file system space.

I suggest you raise an AIX PMR (now called a Support Case) and prepare a snap so AIX support can check your AIX version levels to see if your AIX are missing fixed in this area. You should be running the latest technology level and service pack. Running an older AIX version means you are running AIX with problems that have already been fixed, possibly years ago.

Then they may ask for a perfpmr for normal running and the stuck time.

Of course, malloc is the process requesting more virtual memory.

Does your AIX system run out of real memory and virtual memory (paging space) over time?

Check your AIX logs: errpt -l

Can you add more memory? This might stop the problem or postpone it for a longer time.

Hopefully, you are not running AIX with less than 16 GB? You might be able to boot AIX in 4GB of RAM, but it is not sensible.

Can you restart the program every hour to avoid the problem?

Do you have the application source code?

• At the end of the day AIX Support are not there to rewrite duff code.

Hopefully, something here might help you.

My C++ application is shown as running in the output of ps -ef command but it does not show up in topas. This application has been compiled on AIX 7.1 power6 and is running on AIX 7.2 power9. It runs for few hours or sometimes days and then gets stuck. There are no logs being printed. On further analysis i do see that all the threads that are supposed to be active are in blocked in user state and waiting state in Kernel in the malloc_y call.