Cory,

This is always a challenge, not only the reservation piece but the memory limit part, which should always be higher.  We track such things in RTM pretty heavily at various levels, even organizationally.  With those time series charts, management can push their teams to have better quality reservations.  Of course there is also the ML approach, which I believe IBM sells a product to assist you with.  It's called Predictor I think.  It will use Machine Learning to 'estimate' what a jobs' reservation will be, and when activated, take over the memory reservations.  Kind of a nice topic.  We do the same where we are, but wrote our own system.

When you tighen the screws too much, you then can run into OOM issues.  IBM has a nice feature where it will automatically kill the job abusing their memory reservations the most in order to save the host.  As Sun Yi mentioned, you can always call the IBM support organization to press for additional details on such things.

Larry

Thanks Yi and Larry for the replies. We have RTM tracking in place, and when this problem arises again I will try to compare RTM stats with memory usage reported directly by the linux VM. It's still possible that the problem HPC jobs are actually using the reported amount of memory and LSF is doing what it should. I posted here to see if anyone had experienced errors in LSF measuring the job memory usage. So far, I'm not hearing of anyone seeing that problem.

Cory

Cory,

This is always a challenge, not only the reservation piece but the memory limit part, which should always be higher.  We track such things in RTM pretty heavily at various levels, even organizationally.  With those time series charts, management can push their teams to have better quality reservations.  Of course there is also the ML approach, which I believe IBM sells a product to assist you with.  It's called Predictor I think.  It will use Machine Learning to 'estimate' what a jobs' reservation will be, and when activated, take over the memory reservations.  Kind of a nice topic.  We do the same where we are, but wrote our own system.

When you tighen the screws too much, you then can run into OOM issues.  IBM has a nice feature where it will automatically kill the job abusing their memory reservations the most in order to save the host.  As Sun Yi mentioned, you can always call the IBM support organization to press for additional details on such things.

Larry

Cory,

As a rule, on the more modern OS' starting with RHEL7++, you should always used CGROUP accounting in LSF.  It's much more accurate.  Also, it's imperative that you do NOT include disk cache in the CGROUP metric, which is actually an LSF setting.  If you include Disk Cache, the jobs will crash even though they are not using an RSS value that should otherwise allow them to keep running.  Many years ago, that was the design, and they introduced a setting to ignore CACHE memory in the Memory total for the job.

Thanks Yi and Larry for the replies. We have RTM tracking in place, and when this problem arises again I will try to compare RTM stats with memory usage reported directly by the linux VM. It's still possible that the problem HPC jobs are actually using the reported amount of memory and LSF is doing what it should. I posted here to see if anyone had experienced errors in LSF measuring the job memory usage. So far, I'm not hearing of anyone seeing that problem.

Cory

Cory,

This is always a challenge, not only the reservation piece but the memory limit part, which should always be higher.  We track such things in RTM pretty heavily at various levels, even organizationally.  With those time series charts, management can push their teams to have better quality reservations.  Of course there is also the ML approach, which I believe IBM sells a product to assist you with.  It's called Predictor I think.  It will use Machine Learning to 'estimate' what a jobs' reservation will be, and when activated, take over the memory reservations.  Kind of a nice topic.  We do the same where we are, but wrote our own system.

When you tighen the screws too much, you then can run into OOM issues.  IBM has a nice feature where it will automatically kill the job abusing their memory reservations the most in order to save the host.  As Sun Yi mentioned, you can always call the IBM support organization to press for additional details on such things.

Larry