IBM Spectrum Computing Group

Expand all | Collapse all

Configuring LSF for hierarchical resource allocation

  • 1.  Configuring LSF for hierarchical resource allocation

    Posted Thu June 11, 2020 10:53 AM
    Hi All, We are trying to configure LSF to allow hierarchical resource allocation in our pool. For example we would like to be able to configure something like:



    Such that LSF jobs would be submitted to, for example:
    /snackycake/tasting/amami
    /thermonuclear/testing


    The theory being that the top-level allows executives to make high-level allocation decisions and as you dive into the tree allocation decisions can be made on a more fine-grained basis. But crucially, unused allocation rolls up the tree and is available at each intermediate node before being available to leaves. For example if /snackycake/tasting/salt doesn't use its allocation, that is available to sweet, sour, bitter, and amami before it's made available to /snackycake/randd. If it's not used at that level, it is then available to the /thermonuclear hierarchy in the proportions defined therein.

    I previously used a proprietary batch system at a large company that used that kind of system to manage millions of cores, but have so far not found an equivalent capability in LSF. Job groups seem to get me part-way there but there are several problems:

    * I don't see a way to disallow implicit creation of job groups
    * Job groups appear to only allow specific job-count limits, not percentage-based
    * It's not clear that other branches of the tree can utilize unused job slots


    Any suggestions for what to look for to achieve this?

    Thanks!
    Sam



  • 2.  RE: Configuring LSF for hierarchical resource allocation

    Posted Fri June 19, 2020 12:08 PM
    You may review LSF Fairshare and Guaranteed SLA scheduling policies.
    https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_admin/chap_fairshare_lsf_admin.html
    https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_admin/chap_guar_res_pool.html

    Regards,

    Yi Sun

    ------------------------------
    YI SUN
    ------------------------------