High Performance Computing

 View Only

Are you on the LSF naughty list? Predicting Resource Usage

By Bill McMillan posted Mon December 21, 2020 04:39 PM


With the holiday season once again upon us, we've placed a some presents under the virtual tree.

In many cases the boxes we use to package things are much to big for the items inside. It's disappointing to open a huge box to find something tiny inside - but even more disappointing to be told by the shipper that they couldn't fit the oversized box in the van for delivery. Be Scrooge, use only what is needed - with the right sized box we'd fit more in the van, use less packaging and be nicer to the environment.

The same is often true of the resources we request for running workloads. For example, if users significantly over estimate the amount of memory required by jobs, you end up with fewer jobs per server, lower utilization and longer time to results. If you are running on the Cloud, then you could be paying for much more expensive instance types than you actually need.

IBM Spectrum LSF Predictor is a new offering that utilizes IBM AutoAI to learn about workloads and predict their resource requirements. These AI models can be tested agains the IBM Spectrum LSF Simulator to gain confidence in their use before deploying into production.

LSF Predictor

We've also placed Service Pack 11 for the LSF Family under the tree. Enhancements in this Service Pack include:

  • New Cloud Resource Connectors for Red Hat OpenShift and IBM Cloud Gen2, enabling these to be resource providers when bursting to the cloud.
  • Container runtime support has been extended to Podman and enroot, and cgroup v2 has been enabled.
  • The sample web services python client has been updated to python 3, and is available from our public github.
  • GPU Resource Collection has been optimized, and built-in GPU support extended to support AMD GPU's.
  • Support for Dynamically managing Multi Instance GPU's (MIG) on NVIDIA A100 GPU's will also be under the tree as a separate package, and we'll talk about that in more detail in another blog.

Service Pack 11 is available now from Fix Central