IBM Software Hub allows administrators or users with Monitor Platform permission to set and enforce quotas on valuable resources like vCPU and Memory ensuring that the cluster resources are allocated mindfully. There are two options when setting quotas for CPU and Memory - users can either set quotas on Limits or Requests. Since IBM Software Hub platform can host multiple services like watsonx.ai, Db2 etc. and with multiple projects configured within each of these services, users can decide whether they want to set the CPU / Memory quotas on a Platform level, Service level or Project level.
With the release of IBM Software Hub 5.2.0, users will now be able to set quotas on GPU.
Note: only nvidia.com/GPU is supported.
From the IBM Software Hub Monitoring Page, users can set quota and threshold values for GPU. Kindly note that incase of GPU quotas, the requests and limits mean the same. To set quotas visit the main monitoring dashboard.
On the Platform Resource Usage card you can set GPU Quota for the platform. To set quotas on any service or project, you can select the appropriate option from the Quotas Card.
Alerts and Events for Quota
There are two monitors `check-quota-status` and `check-workload-quota-status` that would generate alerts when the pods GPU requests goes beyond the set threshold.
Suppose you set quota on GPU requests to 8 gpu and the threshold is 90%. User will receive a warning alert when the GPU Requests reaches 90% of 8 gpu i.e. when it reaches 7.2 gpu.
User will receive a critical alert when the GPU requests goes beyond 8 gpu.
Historical Usage graph
Users can also view the Historical Graph for the platform, for a particular service or for a specific project.
Historical Usage graph provides a visual representation of resources and shows how GPU Requests, Quota and Threshold values have changed over time. It helps in understanding trends, patterns, and correlations in the data, which can be useful for various purposes.
IBM SOFTWARE HUB CONTROL CENTER
GPU quota is also added for the IBM Software Hub Control Center. While adding Software Hub Instance to an Account, account admin can set GPU quota for the instance in addition to CPU and Memory Quota.
Important Notes :
- For Enforcement of the GPU Quotas, users can select the Enforce Quotas option. This option is enabled only when CPD Scheduler is installed in the cluster. The scheduler restricts the pods from requesting GPU beyond the set quota value.
- To edit or set quotas, user will need the Administrator or Monitor Platform permission.