High Performance Computing Group

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?

What version of LSF is in use? E.g. run lim -V and sbatchd -V.

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?

We run the latest fix pack (14):

$ lim -V
EGO 3.4.0 build 601547, April 20 2023

$ sbatchd -V
IBM Spectrum LSF 10.1.0.0 build 601547, April 20 2023

binary type: linux3.10-glibc2.17-x86_64

What version of LSF is in use? E.g. run lim -V and sbatchd -V.

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?

Add LSF_GPU_RESOURCE_IGNORE=Y in lsf.conf then restart the cluster. Also make sure you have installed ROCM SMI library on AMD GPU node. 

We run the latest fix pack (14):

$ lim -V
EGO 3.4.0 build 601547, April 20 2023

$ sbatchd -V
IBM Spectrum LSF 10.1.0.0 build 601547, April 20 2023

binary type: linux3.10-glibc2.17-x86_64

What version of LSF is in use? E.g. run lim -V and sbatchd -V.

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?

Thanks!  We will try to add this to lsf.conf.  The ROCM library is installed.

BTW, the documentation about LSF_GPU_RESOURCE_IGNORE is not very clear:  it says "Default: Y", which made us believe it is set to 'Y', if not present in lsf.conf - which is obviously not true! 

Add LSF_GPU_RESOURCE_IGNORE=Y in lsf.conf then restart the cluster. Also make sure you have installed ROCM SMI library on AMD GPU node. 

We run the latest fix pack (14):

$ lim -V
EGO 3.4.0 build 601547, April 20 2023

$ sbatchd -V
IBM Spectrum LSF 10.1.0.0 build 601547, April 20 2023

binary type: linux3.10-glibc2.17-x86_64

What version of LSF is in use? E.g. run lim -V and sbatchd -V.

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?

Thanks!  We will try to add this to lsf.conf.  The ROCM library is installed.

BTW, the documentation about LSF_GPU_RESOURCE_IGNORE is not very clear:  it says "Default: Y", which made us believe it is set to 'Y', if not present in lsf.conf - which is obviously not true! 

Add LSF_GPU_RESOURCE_IGNORE=Y in lsf.conf then restart the cluster. Also make sure you have installed ROCM SMI library on AMD GPU node. 

We run the latest fix pack (14):

$ lim -V
EGO 3.4.0 build 601547, April 20 2023

$ sbatchd -V
IBM Spectrum LSF 10.1.0.0 build 601547, April 20 2023

binary type: linux3.10-glibc2.17-x86_64

What version of LSF is in use? E.g. run lim -V and sbatchd -V.

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?

To be on the safe side, we added the parameter to lsf.conf and restarted the cluster.  Still no success! 

Is there any guide/documentation, what one should be aware of, when using AMD GPUs?  How does LSF find them, and where does it look for it? 

Thanks!  We will try to add this to lsf.conf.  The ROCM library is installed.

BTW, the documentation about LSF_GPU_RESOURCE_IGNORE is not very clear:  it says "Default: Y", which made us believe it is set to 'Y', if not present in lsf.conf - which is obviously not true! 

Add LSF_GPU_RESOURCE_IGNORE=Y in lsf.conf then restart the cluster. Also make sure you have installed ROCM SMI library on AMD GPU node. 

We run the latest fix pack (14):

$ lim -V
EGO 3.4.0 build 601547, April 20 2023

$ sbatchd -V
IBM Spectrum LSF 10.1.0.0 build 601547, April 20 2023

binary type: linux3.10-glibc2.17-x86_64

What version of LSF is in use? E.g. run lim -V and sbatchd -V.

Hi,

we added a new host with two AMD GPUs to our cluster - but they are not detected by LSF!  Is that supported at all by LSF?  We can find a  lot of references to AMD GPUs in the documentation for bsub, etc, but nothing for how to configure LSF to support/detect AMD GPUs.  We use

LSB_GPU_NEW_SYNTAX=extend
LSF_GPU_AUTOCONFIG=Y

in lsf.conf. 

Any hints, what we are missing?