High Performance Computing Group

 View Only
Expand all | Collapse all

CPU-cores max limit per queue

  • 1.  CPU-cores max limit per queue

    Posted Wed December 16, 2020 04:34 PM
    hi team :)

    We're working with IBM Spectrum LSF Standard 10.1.0.6
    I'm supposed to split the CPU-cores ressources between two queues.
    for instance, my queues are hosted on 2 machines 16 CPU-cores and I would like to limit to 10 cores max on the QUEUE-A and 22 cores max for the QUEUE-B.

    Is it possible and how ?
    I found PROCLIMIT but this is deprecated in 10.1 
    https://www.bsc.es/support/LSF/9.1.2/lsf_admin/index.htm?processors_allocated_limiting.html~main
    Bsc remove preview
    View this on Bsc >





    Bsc remove preview
    View this on Bsc >



    ------------------------------
    Stephane colas
    ------------------------------

    #SpectrumComputingGroup


  • 2.  RE: CPU-cores max limit per queue

    Posted Wed December 16, 2020 08:56 PM

    You may consider to use resource limit configuration in lsb.resources file.

    https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_config_ref/lsb.resources.5.html



    ------------------------------
    YI SUN
    ------------------------------



  • 3.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 02:48 AM

    Yes I think so but cpulimit is a number of minutes. 
    so which limit recommend you ?



    ------------------------------
    Stephane colas
    ------------------------------



  • 4.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 03:12 AM

    The magic word you are looking for is SLOTS. 



    ------------------------------
    Bernd Dammann
    ------------------------------



  • 5.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 04:56 AM

    Hi Bernd

    SLOTS. why not but reading the documentation it says that SLOTS is job slots not CPU/core related.

    I know the slots with the lsb.hosts MJX parameter but not this SLOTS parameter in ressources or queue part.

    Could you help me more ?



    ------------------------------
    Stephane colas
    ------------------------------



  • 6.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 05:56 AM

    Somewhere in your configuration you must have defined how LSF should schedule jobs, e.g. "PARALLEL_SCHED_BY_SLOT = Y", and you also need to define somewhere, that a slot corresponds to a core or to a hw-thread.  I don't have access to the documentation right now, but maybe this gives you an idea.



    ------------------------------
    Bernd Dammann
    ------------------------------



  • 7.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 10:12 AM

    Not cpu time limit, as Bernd said it should be SLOT. Here are examples of limits you can set.

    Begin Limit

    NAME=limit1

    SLOTS=10

    QUEUES=queue-A

    End Limit

    Begin Limit

    NAME=limit2

    SLOTS=22

    QUEUES=queue-B

    End Limit

    To match SLOT exactly to CPU core, in lsb.hosts file set following in host section.

    Begin Host
    HOST_NAME    MXJ   ...   
    Default      !     ...
    End Host


    ------------------------------
    YI SUN
    ------------------------------



  • 8.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 03:44 PM

    Oh I see. you link the number of slots to the number of cores ok. 
    I'm sorry but I disagree because this assumption reduces the number of batch execution or interactive sessions. With a 10 cores machine you can run more than 10 interactive sessions for instance without pending sessions. 
    MJX is not ! but 150 for each machine. Do you see ?

    Is there another proposition ?



    ------------------------------
    Stephane colas
    ------------------------------



  • 9.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 04:18 PM

    If you use the schedule by slot setting in lsb.params, you can set the per host scheduling slots as large as you want it. I have customers that set it to 1000, and the only thing that controls scheduling is memory reservations. It works perfectly. Now, if you use Affinity "bsub -R 'afinnity[code|thread|socket|numa]', well then there is no good way of oversubscribing the box.

    Regards,

    Larry Adams
    Global Architect Spectrum Computing

    Global Spectrum RTM Product Lead

    IBM Spectrum Computing


    Mobile: 586-899-1138
    E-mail:
    adamsla@us.ibm.com
    Chat:

    Find me on:
    IBM

    4224 Bold Meadows
    Oakland Township, MI 48306
    United States







  • 10.  RE: CPU-cores max limit per queue

    Posted Thu December 17, 2020 04:33 PM

    Hi Larry 

    not sure to understand  

    if you have a grid environment that host test and prod queues on few machines and people want to limit the test queue cpu usage to 10% of the environment how do you do that ?



    ------------------------------
    Stephane colas
    ------------------------------



  • 11.  RE: CPU-cores max limit per queue

    Posted Mon June 21, 2021 12:19 PM

    A workaround is to define a shared resource, then set limit for it on queue, specify rusage on job.

    lsf.shared
    Begin Resource
    RESOURCENAME TYPE INTERVAL INCREASING CONSUMABLE DESCRIPTION # Keywords
    ...
    cpucores Numeric () N () (CPU cores)
    End Resource

    lsf.cluster
    Begin ResourceMap
    RESOURCENAME LOCATION
    cpucores (32@[hostA hostB])
    End ResourceMap

    lsf.resources
    Begin Limit
    QUEUES RESOURCE
    queueA ([cpucores,10])
    queueB ([cpucores,22])
    End Limit

    -----
    bsub -q queueA -R "rusage[cpucores=5]" ...



    ------------------------------
    Guang Bo Shu
    ------------------------------



  • 12.  RE: CPU-cores max limit per queue

    Posted Mon June 21, 2021 01:57 PM
    Another approach is to use Guarantee Pools.  They provide a nice policy that allows you to delegate not only CPU's but more importantly MEMORY to the various consumer groups, including queues.  This is what I recommend to all customers these days.

    ------------------------------
    Larry Adams
    ------------------------------