How to share resources across clusters?

In environments with multiple LSF clusters, sometimes you need a resource to be shared across clusters. The most common case is floating software licenses. Though potentially the resource could be anything, e.g. user quotas on cores, GPUS, etc.

There are a few standard ways to handle such resources.

Deploy a separate ELIM per cluster

Each cluster runs its own External LIM (ELIM) script which is responsible to publish resource capacities to the cluster. 

In the case of software licenses, typically the ELIM of each cluster will poll license servers to detect the number of idle licenses. This capacity is tracked as a custom resource in LSF. Jobs that need a license will request to LSF to use the custom resource. Such jobs will specify a "reservation duration" of several seconds to a few minutes, during which LSF internally reserves the licenses for the job. This reservation duration is intended to allow the job time to check out its licenses, and for the ELIM to pick up the new license availability. The result is that dispatched jobs don't need to compete with each other for licenses.

While this approach works well in a single cluster, it is a different story when you have multiple clusters. Both ELIMs and the reservations made are internal to each cluster. So it is possible for multiple clusters to see licenses as available and independently decide to dispatch jobs that request them. The result is that running jobs are forced to compete with each other to check out licenses. The result is either failing jobs, or jobs that queue up on the license server while occupying compute resources in the cluster.

Even with a single cluster, it can be challenging to set the reservation duration for a job properly. Too short, and the reservation may expire before the job has a chance to check out its licenses. Too long, and LSF continues to reserve licenses even after the job as checked out its licenses, lowering license utilization.

LSF License Scheduler

License Scheduler is a specialized component of LSF to help distribute floating software licenses among jobs. It provides end-to-end license tracking and enforcement across clusters, simplifying things especially in a multi-cluster environment.

Instead of having each cluster independently determine license capacity, License Scheduler provides a central component to track capacity. It is also responsible to match license checkouts to jobs, and to distribute license capacity among clusters and projects according to configurable policies.

Since License Scheduler can correlate license requests and license checkouts, the there is no need to use the technique of reserving licenses for a short periods, eliminating some complexity and providing benefit even for an environment with a single cluster.

If you have only a single cluster, you can use License Scheduler Basic Edition, which is included with LSF, to collect the license information so that you don't need to write an ELIM.

Configure global resources

While ELIMs and License Scheduler have been a part of LSF for a long time, global resources (along with global limits) is a newer feature. It allows you to define shared capacity that is enforced across clusters. The capacity can be either defined statically in the configuration file, or else dynamically reported as using a special global resource ELIM.

Global resource policies

Global resources are governed by policies that decide how clusters share them. There are two policies available: compete and even distribution.

Compete (default and recommended)

Clusters compete dynamically for global resources based on demand and policy settings. This ensures that resources flow to where they are needed most, maximizing utilization and throughput in dynamic environments.

• Default behavior in LSF global policies
• Recommended because it adapts naturally to workload fluctuations
• Prevents overcommitment while allowing demand-driven allocation

Even distribution

The global resource capacity is split evenly between clusters. While the allocation will be adjusted over time based on each cluster’s usage, the focus is more on sharing evenly between clusters rather than maximizing utilization of the global resource. Consider it only if you have strict partitioning requirements.

Below, I focus only on the compete policy.

How the compete policy works

Under the compete policy, coordination happens between three daemons:

    mbatchd – The central job management daemon (per cluster)
    Maintains local cluster state (pending/running jobs, local resources) and receives dispatch decisions from mbschd

    mbschd – The main LSF scheduler daemon (per cluster).
    Makes placement and dispatch decisions for jobs

    gpolicyd – The global policy daemon (shared/global)
    Tracks global resource availability and usage across clusters

The mbschd within each cluster performs a two-phase sync with gpolicyd.

Pre-scheduling sync   

At the start of each scheduling cycle, mbschd requests the current available capacity for global resources from gpolicyd. Using this capacity snapshot, mbschd evaluates the jobs in its queues and makes tentative placement decisions. Since the schedulers from all clusters are working independently, there is a possibility that resource capacities have changed since the original sync with gpolicyd.

Pre-dispatch validation   

Before dispatching jobs, mbschd performs a second sync with gpolicyd. It sends its intended dispatch decisions to gpolicyd and in response gpolicyd updates its resource usage counters, and replies with which jobs are allowed and which must be withheld to avoid overcommitment. mbschd then cancels any disallowed dispatch decisions before passing the final dispatch list to mbatchd.

Configure a global resource

Below, I give the steps to configure a global resource. The capacity is reported by a global version of the ELIM (called a gres). Note that this global ELIM reports the total capacity for the resource instead of current availability. Therefore, I don't need to specify reservation durations for my jobs.

Configure the global policy daemon

Choose a cluster on which to host the global policy daemon. Set the following parameters in lsf.conf for every cluster that will participate.

LSB_GPD_PORT=7870
LSB_GPD_CLUSTER="sage"

Restart the daemons on the LSF management hosts of all clusters to pick up this change.

Write global ELIM

In the cluster where the global policy daemon runs, run the following to create a global ELIM in $LSF_SERVERDIR. You will need the appropriate permissions.

echo "while [ 1 ]
do
  echo 1 global_res1 10
  sleep 1
done" > $LSF_SERVERDIR/gres ; chmod a+x $LSF_SERVERDIR/gres

Try running this global ELIM from the command line.

$ $LSF_SERVERDIR/gres
1 global_res1 10
1 global_res1 10
1 global_res1 10
...

Configure the global resource

In the head cluster, configure the following in the lsb.globalpolicies file.

Begin Resource
RESOURCENAME    TYPE            INTERVAL        INCREASING      CONSUMABLE      RELEASE         DESCRIPTION
global_res1     Numeric         10              N               Y               Y               (my global resource)
End Resource

Begin ResourceMap
RESOURCENAME    LOCATION
global_res1     ([all])
End ResourceMap

Restart the management daemons on the cluster.

Test it

You can query the global policy daemon directly to see that the resource capacity is being captured properly.

$ bgpinfo resource
RESOURCE                  TOTAL        RESERVED
global_res1                10.0             0.0

You can also check the availability of the resource from each cluster.

$ bhosts -s
RESOURCE                 TOTAL       RESERVED       LOCATION
global_res1               10.0            0.0       ALL

Initially, all clusters will see 10.0 units of the resource until the TOTAL (i.e. current availability) column.  When a unit of the resource allocated to a job in one cluster, all clusters will see this available capacity decrease.

Submit jobs that request this resource from multiple clusters.

$ bsub -R "rusage[global_res1=1]" sleep 10000
Job <4070> is submitted to default queue <normal>.

You should verify that jobs cannot exceed the resource capacity.

Considerations

The compete policy, backed by the two-phase sync between mbschd and gpolicyd, provides a reliable and efficient mechanism to enforce global capacity across LSF clusters. However, be aware that it does not ensure balanced distribution of resource usage across clusters. 

• Potential resource starvation in some clusters. If Cluster A is consuming a global resource heavily, jobs in Cluster B may struggle to dispatch until capacity is freed.
• Scheduler Speed Matters. Clusters with faster scheduling cycles (shorter intervals, more frequent syncs) can have a practical advantage, grabbing resources sooner than slower clusters.

Avoiding starvation for global resources is one area that may warrant further enhancement in LSF.