Hi all,

Needing some help please with IBM Spectrum Platform LSF 10.1.0.10 (plan to install fix pack 14 in the next maintenance window).

Using the following example, host1 has 300GB, if we submit a job using: bsub -n 4 -R "rusage[mem=20000/job]" -R "span[stripe]" -R "hname==host1"  sleep 60, and with RESOURCE_RESERVE_PER_TASK=Y (confdir/lsb.params) the scheduler _initially_ wants to allocate 80000 (80GB)  mem (4x20GB), this can be seen by watching  mem resource using bhosts -l (linux watch), curiously 10 seconds into execution mem drops back to 20000 (20GB). The main problem is If we increase -R "rusage[mem=200000/job]" (extra 0 i.e. 200GB) the job will PEND as it cannot initially allocate 800GB (200GBx4) even though the job only needs 200GB to run.

If we modify RESOURCE_RESERVE_PER_TASK=N then the problem does not occur. It appears that the scheduler initially wants to allocate memory per task as per RESOURCE_RESERVE_PER_TASK=Y, initially ignoring job RES_REQ mem=xxx/job (per job) until upto 10 seconds into execution, this initial per task/per job mismatch can cause jobs to incorrectly PEND.

Please advise is this a bug?

Many Thanks,

Colin 

For parallel job cross nodes, rusage[mem/job] is ambiguous  as LSF doesn't know the memory should be reserved on each node. Or you may argue that reserving memory on node by # of tasks on it. Then can you use rusage[mem] as you set RESOURCE_RESERVE_PER_TASK=Y, or rusage[mem/task]. For weird behavior you are observing, maybe create a support case for more clarification. 

Hi Yi,

Thank you for replying.

The behaviour observed is for a single host, in a test lab here Fix Pack 10 (build 545500) added /job or /task sub options for mem resource in the request string. 

Please can you clariffy, our understanding is -R "rusage[mem=20000/job]" should override RESOURCE_RESERVE_PER_TASK=Y?

Many Thanks,

Colin

Same problem observed with Fix Pack 14 (601547)

Original Message:
Sent: Tue April 29, 2025 07:16 AM
From: Colin Rudakiewicz
Subject: mem=xxx/job (per job) resource request string problem with RESOURCE_RESERVE_PER_TASK=Y

Hi Yi,

Thank you for replying.

The behaviour observed is for a single host, in a test lab here Fix Pack 10 (build 545500) added /job or /task sub options for mem resource in the request string. 

Please can you clariffy, our understanding is -R "rusage[mem=20000/job]" should override RESOURCE_RESERVE_PER_TASK=Y?

Many Thanks,

Colin

------------------------------
Colin Rudakiewicz
------------------------------


Original Message:
Sent: Mon April 28, 2025 05:27 PM
From: YI SUN
Subject: mem=xxx/job (per job) resource request string problem with RESOURCE_RESERVE_PER_TASK=Y

For parallel job cross nodes, rusage[mem/job] is ambiguous  as LSF doesn't know the memory should be reserved on each node. Or you may argue that reserving memory on node by # of tasks on it. Then can you use rusage[mem] as you set RESOURCE_RESERVE_PER_TASK=Y, or rusage[mem/task]. For weird behavior you are observing, maybe create a support case for more clarification. 

------------------------------
YI SUN

In LSF 10.1.0.14, this is considered a limitation in how the scheduler's initial reservation phase handles the /job modifier when forced into per-task mode. While the /job syntax is designed to override per-task defaults, a global setting of RESOURCE_RESERVE_PER_TASK=Y can cause the scheduler to prioritize the task-based calculation during the host selection phase.

The cluster-wide RESOURCE_RESERVE_PER_TASK parameter is actually considered obsolete in favor of the more granular ReservationUsage section in lsb.resources. You can define specifically how memory should be reserved without affecting other resources.<sources-carousel-inline ng-version="0.0.0-PLACEHOLDER" _nghost-ng-c3082671168=""></sources-carousel-inline><mat-icon _ngcontent-ng-c3287480378="" role="img" fonticon="link" class="mat-icon notranslate symbol gds-icon-s google-symbols mat-ligature-font mat-icon-no-color ng-star-inserted" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="link"></mat-icon>

Action: Add the following to your lsb.resources:

<response-element class="" ng-version="0.0.0-PLACEHOLDER"></response-element>

Begin ReservationUsage
RESOURCE   METHOD     RESERVE
mem        PER_JOB    Y
End ReservationUsage

<response-element class="" ng-version="0.0.0-PLACEHOLDER"></response-element>

This forces the scheduler to always treat memory as a per-job resource during the reservation phase, regardless of the number of tasks, which should resolve the incorrect PEND status.