Yes, let's unpack and analyze the question and the answer to clarify the situation and possible solutions.

❓ Question Summary:

• You have a host with multiple GPUs (e.g., 8).

• You want to restrict LSF (IBM Spectrum LSF) from using specific GPUs (e.g., GPU #4 or GPU #6 and #7).

• LSF can restrict the number of GPUs used on a host but not which specific ones.

• So, can you configure LSF to exclude specific GPUs, not just limit the total number?

✅ Key Facts about LSF and CUDA GPU Handling:

1. LSF GPU Scheduling Basics:

   • LSF can schedule GPUs using the resource management system.

   • You can control GPU usage through CUDA_VISIBLE_DEVICES, which LSF sets automatically when managing GPU resources.

   • LSF doesn't inherently know about GPU topology or health - it sees GPU slots as abstract resources (e.g., gpu).

2. Limiting Total GPUs on Host:

   • You can configure the number of GPUs available per host using lsf.conf or host-level resource settings.

   • Example:

     pgsql
     ΑντιγραφήΕπεξεργασία
     Begin Resource RESOURCENAME  TYPE     INTERVAL  DURATION  DESCRIPTION gpu           Boolean  60        60        (gpu) End Resource  Begin Host HOST_NAME   MXJ   ncpus  ngpus hostA       8     16     6 End Host 

   • This restricts the total number of GPUs available for jobs but not specific ones.

�� Answer Analysis (Rephrased):

"You can configure LSF to use only 7 of 8 GPUs, but not configure it to exclude a specific GPU like GPU #4."

This is accurate: LSF itself cannot directly specify which GPU indexes (e.g., 0–7) are to be used or excluded. It just counts GPUs as generic resources.

��️ How to Exclude Specific GPUs: Workarounds

To exclude specific GPUs (e.g., GPU #4), you can use host-level workarounds in combination with LSF's environment variable handling:

✅ Option 1: Use cgroups and NVIDIA GPU visibility restrictions

• You can configure the system or job environment to hide specific GPUs from the driver using:

  bash
  ΑντιγραφήΕπεξεργασία
  export CUDA_VISIBLE_DEVICES=0,1,2,3,5,6,7  # Exclude GPU #4 (which is index 4) 

• You could script this into LSF's job prolog or enforce it via job submission wrappers.

✅ Option 2: Configure a Node as Having Fewer GPUs

• Modify the number of GPUs seen by LSF:

  • Create a wrapper for LSF GPU detection (e.g., override /usr/bin/nvidia-smi or use nvidia-smi topo -m).

  • Create virtual GPU pools on the host using nvidia-container-toolkit or nvidia-smi with MIG (on supported GPUs).

  • These methods can mask specific devices from the system or container environment.

✅ Option 3: Custom CUDA_VISIBLE_DEVICES via bsub

You can override what LSF sets using a submission command:

bash
ΑντιγραφήΕπεξεργασία
bsub -gpu "num=6" -env "CUDA_VISIBLE_DEVICES=0,1,2,3,5,6" my_script.sh 

This tells your job which GPUs to use explicitly.

⚠️ Important: LSF will still think it's using all 6 GPUs, but your job will only "see" those specific ones.

✅ To the Question: "Is there an option to configure it?"

Short Answer:
LSF does not natively support excluding specific GPUs by ID/index, but you can work around it by masking GPUs using:

• CUDA_VISIBLE_DEVICES in job submission or prolog

• cgroups or device-level visibility settings

• node-level configuration (setting ngpus lower, or using MIG on Ampere+ GPUs)

✅ For the Answer: "We want to tell LSF not to use specific two of them." - is it possible?

Directly, no. Indirectly, yes, using the same workarounds.

�� Final Summary