High Performance Computing Group

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

Hi Yi Sun, 

One node job can get GPU allocation on host2... The lsload is as below 

lsload -gpu
HOST_NAME   status ngpus gpu_shared_avg_mut gpu_shared_avg_ut ngpus_physical
host1     ok     4                 0%                0%              4
host2     ok     4                 0%                0%              4

When I tried 

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

I am getting no suitable hosts as below

Job <232873>, User <abc>, Project <default>, Status <PEND>, Queue <cq>, C                     ommand <module load cuda; ./maya2.sh>Sun Mar  3 23:32:59: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>; PENDING REASONS: There are no suitable hosts for the job; SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00/host] span[ptile=1] Effective: - GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: -

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

Seems you should submit test job to queue "gq" rather "cq"

Hi Yi Sun, 

One node job can get GPU allocation on host2... The lsload is as below 

lsload -gpu
HOST_NAME   status ngpus gpu_shared_avg_mut gpu_shared_avg_ut ngpus_physical
host1     ok     4                 0%                0%              4
host2     ok     4                 0%                0%              4

When I tried 

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

I am getting no suitable hosts as below

Job <232873>, User <abc>, Project <default>, Status <PEND>, Queue <cq>, C                     ommand <module load cuda; ./maya2.sh>Sun Mar  3 23:32:59: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>; PENDING REASONS: There are no suitable hosts for the job; SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00/host] span[ptile=1] Effective: - GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: -

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

Sorry my bad. This is the output. But somehow it still uses GPUs only from one host. Also  

EXTERNAL MESSAGES:
 MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT
 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N

If you see these are GPUs that are not allocated. All 4 gpus from host2 are running the task but none from host1

$ bjobs -l 232878Job <232878>, User <abcd>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./demo.sh>, Share group charged                     </GPU>Mon Mar  4 09:25:36: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>;Mon Mar  4 09:25:57: Started 2 Task(s) on Host(s) <1*host2> <1*host1>, Al                     located 2 Slot(s) on Host(s) <1*host2> <1*host1>, Ex                     ecution Home </users/analysis/e40070822>, Execution CWD </                     users/analysis/e40070822>;Mon Mar  4 09:28:03: Resource usage collected.                     The CPU time used is 73 seconds.                     MEM: 12.3 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 197 MEMORY USAGE: MAX MEM: 12.3 Gbytes;  AVG MEM: 7.5 Gbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.98 ;  CPU Efficiency: 49.21% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:p                     g] rusage[ngpus_physical=4.00/host] span[ptile=1] Effective: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:                     pg] rusage[ngpus_physical=4.00/host] span[ptile=1] GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

Seems you should submit test job to queue "gq" rather "cq"

Hi Yi Sun, 

One node job can get GPU allocation on host2... The lsload is as below 

lsload -gpu
HOST_NAME   status ngpus gpu_shared_avg_mut gpu_shared_avg_ut ngpus_physical
host1     ok     4                 0%                0%              4
host2     ok     4                 0%                0%              4

When I tried 

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

I am getting no suitable hosts as below

Job <232873>, User <abc>, Project <default>, Status <PEND>, Queue <cq>, C                     ommand <module load cuda; ./maya2.sh>Sun Mar  3 23:32:59: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>; PENDING REASONS: There are no suitable hosts for the job; SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00/host] span[ptile=1] Effective: - GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: -

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

How did you launch the 2nd task on host1?

Sorry my bad. This is the output. But somehow it still uses GPUs only from one host. Also  

EXTERNAL MESSAGES:
 MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT
 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N

If you see these are GPUs that are not allocated. All 4 gpus from host2 are running the task but none from host1

$ bjobs -l 232878Job <232878>, User <abcd>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./demo.sh>, Share group charged                     </GPU>Mon Mar  4 09:25:36: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>;Mon Mar  4 09:25:57: Started 2 Task(s) on Host(s) <1*host2> <1*host1>, Al                     located 2 Slot(s) on Host(s) <1*host2> <1*host1>, Ex                     ecution Home </users/analysis/e40070822>, Execution CWD </                     users/analysis/e40070822>;Mon Mar  4 09:28:03: Resource usage collected.                     The CPU time used is 73 seconds.                     MEM: 12.3 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 197 MEMORY USAGE: MAX MEM: 12.3 Gbytes;  AVG MEM: 7.5 Gbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.98 ;  CPU Efficiency: 49.21% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:p                     g] rusage[ngpus_physical=4.00/host] span[ptile=1] Effective: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:                     pg] rusage[ngpus_physical=4.00/host] span[ptile=1] GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

Seems you should submit test job to queue "gq" rather "cq"

Hi Yi Sun, 

One node job can get GPU allocation on host2... The lsload is as below 

lsload -gpu
HOST_NAME   status ngpus gpu_shared_avg_mut gpu_shared_avg_ut ngpus_physical
host1     ok     4                 0%                0%              4
host2     ok     4                 0%                0%              4

When I tried 

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

I am getting no suitable hosts as below

Job <232873>, User <abc>, Project <default>, Status <PEND>, Queue <cq>, C                     ommand <module load cuda; ./maya2.sh>Sun Mar  3 23:32:59: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>; PENDING REASONS: There are no suitable hosts for the job; SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00/host] span[ptile=1] Effective: - GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: -

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

You may check this link.

https://community.ibm.com/community/user/cloud/discussion/how-to-run-pytorch-ddp-job-on-multi-nodes#bmcfa6563e-098c-4ad0-88f7-a0615a97de40

------------------------------
YI SUN

Original Message:
Sent: Mon March 04, 2024 04:19 PM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

How did you launch the 2nd task on host1?

------------------------------
YI SUN

Original Message:
Sent: Sun March 03, 2024 11:03 PM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

Sorry my bad. This is the output. But somehow it still uses GPUs only from one host. Also  

EXTERNAL MESSAGES:
 MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT
 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N

If you see these are GPUs that are not allocated. All 4 gpus from host2 are running the task but none from host1

$ bjobs -l 232878Job <232878>, User <abcd>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./demo.sh>, Share group charged                     </GPU>Mon Mar  4 09:25:36: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>;Mon Mar  4 09:25:57: Started 2 Task(s) on Host(s) <1*host2> <1*host1>, Al                     located 2 Slot(s) on Host(s) <1*host2> <1*host1>, Ex                     ecution Home </users/analysis/e40070822>, Execution CWD </                     users/analysis/e40070822>;Mon Mar  4 09:28:03: Resource usage collected.                     The CPU time used is 73 seconds.                     MEM: 12.3 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 197 MEMORY USAGE: MAX MEM: 12.3 Gbytes;  AVG MEM: 7.5 Gbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.98 ;  CPU Efficiency: 49.21% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:p                     g] rusage[ngpus_physical=4.00/host] span[ptile=1] Effective: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:                     pg] rusage[ngpus_physical=4.00/host] span[ptile=1] GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

------------------------------
Subin Pillai

Original Message:
Sent: Sun March 03, 2024 03:59 PM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

Seems you should submit test job to queue "gq" rather "cq"

------------------------------
YI SUN

Original Message:
Sent: Sun March 03, 2024 01:05 PM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

Hi Yi Sun, 

One node job can get GPU allocation on host2... The lsload is as below 

lsload -gpu
HOST_NAME   status ngpus gpu_shared_avg_mut gpu_shared_avg_ut ngpus_physical
host1     ok     4                 0%                0%              4
host2     ok     4                 0%                0%              4

When I tried 

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

I am getting no suitable hosts as below

Job <232873>, User <abc>, Project <default>, Status <PEND>, Queue <cq>, C                     ommand <module load cuda; ./maya2.sh>Sun Mar  3 23:32:59: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>; PENDING REASONS: There are no suitable hosts for the job; SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00/host] span[ptile=1] Effective: - GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: -

------------------------------
Subin Pillai

Original Message:
Sent: Thu February 22, 2024 11:53 AM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

------------------------------
YI SUN

Original Message:
Sent: Wed February 21, 2024 03:25 AM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

------------------------------
Subin Pillai
------------------------------

This really helped. I didn't use blaunch. That was the mssing piece of the puzzle. This is the updated command I used.

bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log blaunch -z "host1 host2" "module load cuda; ./file.sh"

But the problem now is it runs the program twice. That is I am getting the outputs twice, instead of it running parallel. 

------------------------------
Subin Pillai

Original Message:
Sent: Mon March 04, 2024 04:24 PM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

You may check this link.

https://community.ibm.com/community/user/cloud/discussion/how-to-run-pytorch-ddp-job-on-multi-nodes#bmcfa6563e-098c-4ad0-88f7-a0615a97de40

------------------------------
YI SUN

Original Message:
Sent: Mon March 04, 2024 04:19 PM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

How did you launch the 2nd task on host1?

------------------------------
YI SUN

Original Message:
Sent: Sun March 03, 2024 11:03 PM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

Sorry my bad. This is the output. But somehow it still uses GPUs only from one host. Also  

EXTERNAL MESSAGES:
 MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT
 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N

If you see these are GPUs that are not allocated. All 4 gpus from host2 are running the task but none from host1

$ bjobs -l 232878Job <232878>, User <abcd>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./demo.sh>, Share group charged                     </GPU>Mon Mar  4 09:25:36: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>;Mon Mar  4 09:25:57: Started 2 Task(s) on Host(s) <1*host2> <1*host1>, Al                     located 2 Slot(s) on Host(s) <1*host2> <1*host1>, Ex                     ecution Home </users/analysis/e40070822>, Execution CWD </                     users/analysis/e40070822>;Mon Mar  4 09:28:03: Resource usage collected.                     The CPU time used is 73 seconds.                     MEM: 12.3 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 197 MEMORY USAGE: MAX MEM: 12.3 Gbytes;  AVG MEM: 7.5 Gbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.98 ;  CPU Efficiency: 49.21% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Mar  4 09:26   host1:gpus=0,1,2,3;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:p                     g] rusage[ngpus_physical=4.00/host] span[ptile=1] Effective: select[((type == any ) && (type == any )) && (ngpus>0)] order[r15s:                     pg] rusage[ngpus_physical=4.00/host] span[ptile=1] GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

------------------------------
Subin Pillai

Original Message:
Sent: Sun March 03, 2024 03:59 PM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

Seems you should submit test job to queue "gq" rather "cq"

------------------------------
YI SUN

Original Message:
Sent: Sun March 03, 2024 01:05 PM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

Hi Yi Sun, 

One node job can get GPU allocation on host2... The lsload is as below 

lsload -gpu
HOST_NAME   status ngpus gpu_shared_avg_mut gpu_shared_avg_ut ngpus_physical
host1     ok     4                 0%                0%              4
host2     ok     4                 0%                0%              4

When I tried 

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

I am getting no suitable hosts as below

Job <232873>, User <abc>, Project <default>, Status <PEND>, Queue <cq>, C                     ommand <module load cuda; ./maya2.sh>Sun Mar  3 23:32:59: Submitted from host <host1>, CWD <$HOME>, 2 Task(s), Re                     quested Resources <type==any span[ptile=1]>, Requested GPU                      <num=4/host>; PENDING REASONS: There are no suitable hosts for the job; SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00/host] span[ptile=1] Effective: - GPU REQUIREMENT DETAILS: Combined: num=4/host:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: -

------------------------------
Subin Pillai

Original Message:
Sent: Thu February 22, 2024 11:53 AM
From: YI SUN
Subject: Accessing multiple GPUs on different hosts using LSF

Try following see if one node job can get GPU allocation on host2.

• On host1, bsub -I -gpu "num=4" -m host2 nvidia-smi

If above test is positive, try following see if it works on two nodes.

• On host1, bsub -n 2 -gpu "num=4/host" -R "type==any span[ptile=1]" blaunch nvdia-smi

Also make sure on lsload -gpu and bhosts -gpu on host2 report correct GPU info.

------------------------------
YI SUN

Original Message:
Sent: Wed February 21, 2024 03:25 AM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

------------------------------
Subin Pillai
------------------------------

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

The bjobs output shows, that there are GPUs allocated on host2 for the job.  The output field is truncated, but it says "host2:gpus", which wouldn't be there, if there were no GPUs allocated. 

Are you sure, that there are the necessary commands in file.sh, such that processes get launched on both nodes?   file.sh is executed on the first node, only!

------------------------------
Bernd Dammann

Original Message:
Sent: Wed February 21, 2024 03:25 AM
From: Subin Pillai
Subject: Accessing multiple GPUs on different hosts using LSF

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts with 4 GPUs each. I want to use all the 8 GPUs together. But the task is allotted to only one host? 

The command i used to schedule the job is as follows :- 

 bsub -q gq -n 96 -gpu "num=4:mode=shared:j_exclusive=no" -W 6:00 -o output2.log -e error2.log "module load cuda; ./file.sh"

The bjobs -l output is as follows :- 

 bjobs -l 229990Job <229990>, User <abc>, Project <default>, Status <RUN>, Queue <gq>, Co                     mmand <module load cuda; ./maya2.sh>, Share group charged                     </GPU>Wed Feb 21 13:43:13: Submitted from host <host1>, CWD <$HOME>, Output File <                     output2.log>, Error File <error2.log>, 96 Task(s), Request                     ed GPU <num=4:mode=shared:j_exclusive=no>;Wed Feb 21 13:43:13: Started 96 Task(s) on Host(s) <48*host1> <48*host2>,                      Allocated 96 Slot(s) on Host(s) <48*host1> <48*host2                     k>, Execution Home </users/sys/abc>, Execution                     CWD </users/sys/abc>;Wed Feb 21 13:44:24: Resource usage collected.                     The CPU time used is 18 seconds.                     MEM: 1 Gbytes;  SWAP: 0 Mbytes;  NTHREAD: 78 RUNLIMIT 360.0 min MEMORY USAGE: MAX MEM: 1 Gbytes;  AVG MEM: 480 Mbytes; MEM Efficiency: 0.00% CPU USAGE: CPU PEAK: 0.25 ;  CPU Efficiency: 0.26% SCHEDULING PARAMETERS:           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem loadSched   -     -     -     -       -     -    -     -     -      -      - loadStop    -     -     -     -       -     -    -     -     -      -      - EXTERNAL MESSAGES: MSG_ID FROM       POST_TIME      MESSAGE                             ATTACHMENT 0      e40070822  Feb 21 13:43   host1:gpus=1,2,3,0;host2:gpus     N RESOURCE REQUIREMENT DETAILS: Combined: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phys                     ical=4.00] Effective: select[(type == any ) && (ngpus>0)] order[r15s:pg] rusage[ngpus_phy                     sical=4.00] GPU REQUIREMENT DETAILS: Combined: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia Effective: num=4:mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

No GPUs from host 2 are allocated to the task. I checked the same using nvidea-smi command also. The task is run only on host1. What should I do so that I can use all 8 GPUs across both the hosts? Is there some mistake in the bsub command I am using??

------------------------------
Subin Pillai
------------------------------