Hi all,

Needing some help please with IBM Spectrum Platform LSF 10.1.0.10 (plan to install fix pack 14 in the next maintenance window).

Using the following example, host1 has 300GB, if we submit a job using: bsub -n 4 -R "rusage[mem=20000/job]" -R "span[stripe]" -R "hname==host1"  sleep 60, and with RESOURCE_RESERVE_PER_TASK=Y (confdir/lsb.params) the scheduler _initially_ wants to allocate 80000 (80GB)  mem (4x20GB), this can be seen by watching  mem resource using bhosts -l (linux watch), curiously 10 seconds into execution mem drops back to 20000 (20GB). The main problem is If we increase -R "rusage[mem=200000/job]" (extra 0 i.e. 200GB) the job will PEND as it cannot initially allocate 800GB (200GBx4) even though the job only needs 200GB to run.

If we modify RESOURCE_RESERVE_PER_TASK=N then the problem does not occur. It appears that the scheduler initially wants to allocate memory per task as per RESOURCE_RESERVE_PER_TASK=Y, initially ignoring job RES_REQ mem=xxx/job (per job) until upto 10 seconds into execution, this initial per task/per job mismatch can cause jobs to incorrectly PEND.

Please advise is this a bug?

Many Thanks,

Colin 

For parallel job cross nodes, rusage[mem/job] is ambiguous  as LSF doesn't know the memory should be reserved on each node. Or you may argue that reserving memory on node by # of tasks on it. Then can you use rusage[mem] as you set RESOURCE_RESERVE_PER_TASK=Y, or rusage[mem/task]. For weird behavior you are observing, maybe create a support case for more clarification. 

Hi all,

Needing some help please with IBM Spectrum Platform LSF 10.1.0.10 (plan to install fix pack 14 in the next maintenance window).

Using the following example, host1 has 300GB, if we submit a job using: bsub -n 4 -R "rusage[mem=20000/job]" -R "span[stripe]" -R "hname==host1"  sleep 60, and with RESOURCE_RESERVE_PER_TASK=Y (confdir/lsb.params) the scheduler _initially_ wants to allocate 80000 (80GB)  mem (4x20GB), this can be seen by watching  mem resource using bhosts -l (linux watch), curiously 10 seconds into execution mem drops back to 20000 (20GB). The main problem is If we increase -R "rusage[mem=200000/job]" (extra 0 i.e. 200GB) the job will PEND as it cannot initially allocate 800GB (200GBx4) even though the job only needs 200GB to run.

If we modify RESOURCE_RESERVE_PER_TASK=N then the problem does not occur. It appears that the scheduler initially wants to allocate memory per task as per RESOURCE_RESERVE_PER_TASK=Y, initially ignoring job RES_REQ mem=xxx/job (per job) until upto 10 seconds into execution, this initial per task/per job mismatch can cause jobs to incorrectly PEND.

Please advise is this a bug?

Many Thanks,

Colin 

Hi Yi,

Thank you for replying.

The behaviour observed is for a single host, in a test lab here Fix Pack 10 (build 545500) added /job or /task sub options for mem resource in the request string. 

Please can you clariffy, our understanding is -R "rusage[mem=20000/job]" should override RESOURCE_RESERVE_PER_TASK=Y?

Many Thanks,

Colin

For parallel job cross nodes, rusage[mem/job] is ambiguous  as LSF doesn't know the memory should be reserved on each node. Or you may argue that reserving memory on node by # of tasks on it. Then can you use rusage[mem] as you set RESOURCE_RESERVE_PER_TASK=Y, or rusage[mem/task]. For weird behavior you are observing, maybe create a support case for more clarification. 

Hi all,

Needing some help please with IBM Spectrum Platform LSF 10.1.0.10 (plan to install fix pack 14 in the next maintenance window).

Using the following example, host1 has 300GB, if we submit a job using: bsub -n 4 -R "rusage[mem=20000/job]" -R "span[stripe]" -R "hname==host1"  sleep 60, and with RESOURCE_RESERVE_PER_TASK=Y (confdir/lsb.params) the scheduler _initially_ wants to allocate 80000 (80GB)  mem (4x20GB), this can be seen by watching  mem resource using bhosts -l (linux watch), curiously 10 seconds into execution mem drops back to 20000 (20GB). The main problem is If we increase -R "rusage[mem=200000/job]" (extra 0 i.e. 200GB) the job will PEND as it cannot initially allocate 800GB (200GBx4) even though the job only needs 200GB to run.

If we modify RESOURCE_RESERVE_PER_TASK=N then the problem does not occur. It appears that the scheduler initially wants to allocate memory per task as per RESOURCE_RESERVE_PER_TASK=Y, initially ignoring job RES_REQ mem=xxx/job (per job) until upto 10 seconds into execution, this initial per task/per job mismatch can cause jobs to incorrectly PEND.

Please advise is this a bug?

Many Thanks,

Colin