Originally posted by: Syler1983

Hi, all.

I'm having a problem with openmpi 2.1.0 and Abinit.
When i run abinit task with np (number of processes) = 1 task executes fine. With np = 2 small deviations appear. With larger np task ends with an error.

I'm working with POWER8 architecture.

So the question is: can the problem be in incompability of openmpi and POWER8? Should i use spectrum mpi instead?

Thanks,
Vladimir.

Originally posted by: Grigor Nikolov

Hi Vladimir,

There is no known issue for OpenMPI on POWER8.
Trying to replicate the problem.
Meanwhile, could you please let us know:
  -- How did you build openmpi 2.1.0 (compilers and options used)
  -- How did you build Abinit (compiler and options used)
  -- How did you run it - if can provide the mpirun command you used?
  -- More about your environment like OS (RHEL 7.3 or Ubintu #)…

Of course, Spectrum MPI is always great and decidedly optimized alternative on POWER8.

Regards,
Grigor

Originally posted by: Syler1983

Hi Grigor,

Thanks for the reply.
I am working with linux-cluster (arch is ppc64le).
OS is Centos 7.
Run command is

mpiexec -np <number> <path to abinit>/abinit < files >& log

Abinit configure options are:

enable_mpi=yes
enable_mpi_io=yes
with_mpi_prefix="/opt/soft/openmpi/2.1.0/gcc/"
CFLAGS_OPTIM="-O2 -mtune=native"
with_linalg_flavor="essl"
with_linalg_libs="-L/opt/ibmmath/essl/5.5/lib64 -lessl"
enable_openmp="yes"
enable_gpu=yes
with_gpu_flavor="cuda-double"
with_gpu_incs="-I/usr/local/cuda/include"
with_gpu_libs="-L/usr/local/cuda/lib64 -lcublas -lcufft -lcudart"

I compiled abinit using GNU C/C++/Fortran 4.8.5 compilers and NVIDIA Cuda 8.0.61

I can't recall exact openmpi compilation flags but can tell openmpi built with standart flags except PBS support.

Originally posted by: Bill_Buros

Any update on this ?      Do we need more eyes?     

Interesting this is being done with Cuda..    is this being used with the GPUs?   Minsky/P100 ?

Originally posted by: Grigor Nikolov

Hi Bill,
I am trying to reproduce the issue using XL compilers - still unsuccessfully. Sorry for the delay.
If possible, could you please try with gcc/gfortran?
 

Originally posted by: BillSchmidt

I am not an expert on OpenMP (nor on CentOS), but GCC 4.8.5 is extremely old and likely to be downlevel enough wrt OpenMP to cause problems.  Normally I would direct you to the Advance Toolchain, but I don't know how well that is supported on CentOS.  I think the RHEL downloads *should* work, but I'm not certain.  The latest AT 10.0 would provide you with GCC 6.3, which I would expect to do better.  You can find download information here:  https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/IBM%20Advance%20Toolchain%20for%20PowerLinux%20Documentation.

Another option is to install Red Hat's Developer Tool Set 6, which will also provide GCC 6 (but will not provide updated libraries).  The OpenMP libs are provided by gcc, though, so this should be fine.