Originally posted by: Andreas Herten

For a student project on one of our machines I'm looking into the offloading capabilities of OpenMP in XL C compiler 16.1.0. The compilation terminates with an error. I have same error with an XL 14.1.0 installation.

I have created a configuration file with

xlc_configure -o config_xlc -gcc /gpfs/software/opt/gcc/5.4.0/ -cuda $CUDA_ROOT -cudaVersion 9.1 -cuda_cc_major 6 -cuda_cc_minor 0

($CUDA_ROOT expands to /gpfs/software/opt/cuda/9.1.85)

When calling xlc with this configuration I get the error below. It looks like XL might be mixing up LLVM/GCC headers.

How is this possible? And how can I fix it?

(The system is comprised of 18 Minsky nodes, that is: two sockets of POWER8 processors, each attached to two Tesla P100 GPUs.)

-Andreas

$ xlc++ -F config_xlc -qsmp=omp -qoffload pi-gpu.c
In file included from /gpfs/software/opt/cuda/9.1.85/include/fatBinaryCtl.h:19:0,
                 from pi-gpu.fatbin.c:2,
                 from pi-gpu.fatbinwrap.c:2:
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:34:19: error: missing binary operator before token "("
 #if !__has_feature(modules)
                   ^
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:46:42: error: missing binary operator before token "("
 #if !defined(_PTRDIFF_T) || __has_feature(modules)
                                          ^
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:57:39: error: missing binary operator before token "("
 #if !defined(_SIZE_T) || __has_feature(modules)
                                       ^
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:71:42: error: missing binary operator before token "("
      !defined(_RSIZE_T)) || __has_feature(modules)
                                          ^
In file included from pi-gpu.fatbin.c:2:0,
                 from pi-gpu.fatbinwrap.c:2:
/gpfs/software/opt/cuda/9.1.85/include/fatBinaryCtl.h:87:62: error: 'size_t' has not been declared
                                                  void* *elf, size_t *esize);
                                                              ^
In file included from pi-gpu.fatbinwrap.c:8:0:
/gpfs/software/opt/cuda/9.1.85/include/crt/host_runtime.h:148:9: error: 'size_t' has not been declared
         size_t size,
         ^
/gpfs/software/opt/cuda/9.1.85/include/crt/host_runtime.h:159:9: error: 'size_t' has not been declared
         size_t size,
         ^
1586-346 (U) An error occurred during code generation.  The code generation return code was 255.

Originally posted by: Rafik_Zurob

Hi Andreas

This is orthogonal to to the __has_feature(modules) issue, but I thought I'd mention it since you'll hit it next:  XL 16.1.0 only supports CUDA 9.2.  We do not support CUDA 9.1.  When configuring the compiler, it's best to let the compiler figure out the toolkit stuff on its own.  For example:

xlc_configure -o config_xlc -gcc /gpfs/software/opt/gcc/5.4.0/ -cuda $CUDA_ROOT

The configuration utility will then automatically figure out the installed toolkit version and will test the first GPU on the machine to find out its architecture level.

As for the __has_feature(modules) issue, we'll need to investigate further.  We regularly run the compiler with GCC 5.4.0 on our machines.  Can you please paste the output from "ls /gpfs/software/opt/gcc/5.4.0/"?

Thanks

Rafik

Originally posted by: Andreas Herten

Hey Rafik!

Concerning CUDA 9.2: I read that in the release notes. Unfortunately, our cluster currently only has 9.1. My hope was to overcome the 9.2 limitation with that until we had the chance to upgrade our CUDA stack. Since our XL 14.1.0 installation (compatible to CUDA 8 as far as I know) has the exact same error message, my hope was that the error was unrelated to the exact CUDA version. Sidenote: I need to specify cuda_cc_{maj|min}or since the node used for compilation does not have a GPU attached.

Following, the output of ls!

-Andreas

# aherten at juron1-adm in ~/TMP/OpenMP-Offloading [21:08:35]
$ ls /gpfs/software/opt/gcc/5.4.0/
bin  include  lib  lib64  libexec  share

# aherten at juron1-adm in ~/TMP/OpenMP-Offloading [21:09:35]
$ ls /gpfs/software/opt/gcc/5.4.0/include
c++

# aherten at juron1-adm in ~/TMP/OpenMP-Offloading [21:09:43]
$ ls /gpfs/software/opt/gcc/5.4.0/bin
aot-compile    gcj         gkeytool       powerpc64le-unknown-linux-gnu-c++
c++            gcj-dbtool  gnative2ascii  powerpc64le-unknown-linux-gnu-g++
cpp            gcjh        gorbd          powerpc64le-unknown-linux-gnu-gcc
g++            gcov        grmic          powerpc64le-unknown-linux-gnu-gcc-5.4.0
gappletviewer  gcov-tool   grmid          powerpc64le-unknown-linux-gnu-gcc-ar
gc-analyze     gfortran    grmiregistry   powerpc64le-unknown-linux-gnu-gcc-nm
gcc            gij         gserialver     powerpc64le-unknown-linux-gnu-gcc-ranlib
gcc-ar         gjar        gtnameserv     powerpc64le-unknown-linux-gnu-gcj
gcc-nm         gjarsigner  jcf-dump       powerpc64le-unknown-linux-gnu-gfortran
gcc-ranlib     gjavah      jv-convert     rebuild-gcj-db

Originally posted by: Rafik_Zurob

Hi Andreas

Thanks for the output.  We'll investigate.

As for the CUDA Toolkit, the 16.1.0 compiler won't work with anything older than CUDA 9.2.  The reason is that we ship our OpenMP library for the device as an LLVM bitcode file (*.bc) to allow inlining.  We built our library with version 9.2 of the CUDA Toolkit.  The CUDA Toolkit programs (device assembler, device linker, device driver, ..., etc) do not accept code from newer toolkits.  So if you have CUDA 9.1 installed, it will complain about our device libraries (built with CUDA 9.2) and stop compilation.

Regards

Rafik

Originally posted by: Andreas Herten

Dear Rafik,

I updated the CUDA Toolkit to 9.2. As expected, the error stays exactly the same.

-Andreas

Originally posted by: Rafik_Zurob

Hi Andreas

I'm having trouble reproducing the error.  I downloaded GCC 5.4.0, built it with --prefix=/scratch1/rzurob/h/usr/local, and did a make install.  I got a similar directory to what you have above:

$ ls /scratch1/rzurob/h/usr/local
bin  include  lib  lib64  libexec  share
$ ls /scratch1/rzurob/h/usr/local/bin
aot-compile    gcj         gkeytool       powerpc64le-unknown-linux-gnu-c++
c++            gcj-dbtool  gnative2ascii  powerpc64le-unknown-linux-gnu-g++
cpp            gcjh        gorbd          powerpc64le-unknown-linux-gnu-gcc
g++            gcov        grmic          powerpc64le-unknown-linux-gnu-gcc-5.4.0
gappletviewer  gcov-tool   grmid          powerpc64le-unknown-linux-gnu-gcc-ar
gc-analyze     gfortran    grmiregistry   powerpc64le-unknown-linux-gnu-gcc-nm
gcc            gij         gserialver     powerpc64le-unknown-linux-gnu-gcc-ranlib
gcc-ar         gjar        gtnameserv     powerpc64le-unknown-linux-gnu-gcj
gcc-nm         gjarsigner  jcf-dump       powerpc64le-unknown-linux-gnu-gfortran
gcc-ranlib     gjavah      jv-convert     rebuild-gcj-db
$

I also put the CUDA 9.2 toolkit in a non-default location to match your non-default CUDA_ROOT above:

$ ls /xl_le/gsa/tlbgsa/projects/x/xlcmpbld/bld_env/rhel7_leppc/CUDA-9.2.88-1/usr/local/cuda-9.2
bin  extras   lib64      libnvvp  nvml  README   share    tools
doc  include  libnsight  LICENSE  nvvm  samples  targets  version.txt
$

I then configured the compiler as follows:

$ ./opt/ibm/xlC/16.1.0/bin/xlc_configure -o config_xlc \
  -gcc /scratch1/rzurob/h/usr/local \
  -cuda /xl_le/gsa/tlbgsa/projects/x/xlcmpbld/bld_env/rhel7_leppc/CUDA-9.2.88-1/usr/local/cuda-9.2 \
  -cuda_cc_major 6 -cuda_cc_minor 0 -nosymlink
$

which was successful.

I think we should try to eliminate the non-default gcc directory as a factor next:  Does the compiler configure successfully if you don't specify the -gcc option?  (That would use the gcc installed on the system)

If so:

1. What is the output from "xlc -qversion=verbose"?

2. Can you try downloading and building gcc 5.4.0 as I did to make sure we have exactly the same thing?

Here are the exact steps I used to download, build, and install gcc 5.4.0 to a non-default location:

mkdir -p /scratch1/rzurob/h/usr/local
cd /scratch1/rzurob/h/

wget 
http://mirror.csclub.uwaterloo.ca/gnu/gcc/gcc-5.4.0/gcc-5.4.0.tar.gz
wget 
ftp://gcc.gnu.org/pub/gcc/infrastructure/gmp-6.1.0.tar.bz2
wget 
ftp://gcc.gnu.org/pub/gcc/infrastructure/mpfr-3.1.4.tar.bz2
wget 
ftp://gcc.gnu.org/pub/gcc/infrastructure/mpc-1.0.3.tar.gz

tar -xzf gcc-5.4.0.tar.gz
tar -xjf gmp-6.1.0.tar.bz2
tar -xjf mpfr-3.1.4.tar.bz2
tar -xzf mpc-1.0.3.tar.gz

cd gmp-6.1.0/
./configure --prefix=/scratch1/rzurob/h/usr/local
make -j 20
make -j 20 check
make install
cd ../mpfr-3.1.4
./configure --prefix=/scratch1/rzurob/h/usr/local --with-gmp=/scratch1/rzurob/h/usr/local
make -j 20
make -j 20 check
make install
cd ../mpc-1.0.3
./configure --prefix=/scratch1/rzurob/h/usr/local --with-gmp=/scratch1/rzurob/h/usr/local --with-mpfr=/scratch1/rzurob/h/usr/local
make -j 20
make -j 20 check
make install
cd ../gcc-5.4.0/
./configure --prefix=/scratch1/rzurob/h/usr/local --with-gmp=/scratch1/rzurob/h/usr/local --with-mpfr=/scratch1/rzurob/h/usr/local --with-mpc=/scratch1/rzurob/h/usr/local
make -j 20
make install

Originally posted by: Andreas Herten

I haven't found the time to work on your suggestions this week. It's on the top of my to do list for next week!

Thank you a lot, though, for your effort!

Originally posted by: Andreas Herten

Dear Rafik,

I include my progress on the individual steps in the following.

xlc_configure w/o -gcc

I called xlc_configure without the -gcc option twice. In the first time, GCC 5.4.0 was still loaded by the module system available on the node, in the second time it was not. I configured xlc with

/gpfs/software/opt/xlc/16.1/xlC/16.1.0/bin/xlc_configure -o config_xlc -cuda /gpfs/software/opt/cuda/9.2.88 -cuda_cc_major 6 -cuda_cc_minor 0 -nosymlink

In both cases, xlc_configure returned no error; but compiling the OpenMP-enabled program did not work.

If I call xlc++ without any OpenMP flags, thus making the OpenMP statements to meaningless pragmas, the program compiles as expected.

-qinfo

I call xlc -qversion=verbose, once with the config which is using the system GCC (4.8.0) and once configured with the 5.4.0 GCC.

1. System GCC
   $ xlc -qversion=verbose -F config_xlc
   IBM XL C/C++ for Linux, V16.1.0 (Community Edition)
   Version: 16.01.0000.0000
   Driver Version: 16.1.0(C/C++) Level: 180412 ID: _cDgp0De0EeiRmNzegUH2Xw
   C/C++ Front End Version: 16.1.0(C/C++) Level: 180412 ID: _d-hUqDz8EeiRm9zegUH2Xw
   High-Level Optimizer Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level: 180412 ID: _rAOlVzzZEeiRm9zegUH2Xw
   Low-Level Optimizer Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level: 180412 ID: _ayqxljzZEeiRm9zegUH2Xw
   Intermediate Language Splitter Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _XXdUwfgWEeed2OHRoHeItA
   W-Code to LLVM-IR Translator: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _20SS0Bc1Eeid4OHRoHeItA
   NVVM-IR to PTX Translator: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _ePggcSCAEeiwhrbSB1AWhQ
   /gpfs/software/opt/xlc/16.1/xlC/16.1.0//bin/.orig/xlc: note: XL C/C++ Community Edition is a no-charge product and does not include official IBM support. You can provide feedback at the XL on POWER C/C++ Community Edition forum (http://ibm.biz/xlcpp-linux-ce). For information about a fully supported XL C/C++ compiler, visit XL C/C++ for Linux (http://ibm.biz/xlcpp-linux).
2. GCC 5.4.0
   $ xlc -qversion=verbose -F config_xlc
   IBM XL C/C++ for Linux, V16.1.0 (Community Edition)
   Version: 16.01.0000.0000
   Driver Version: 16.1.0(C/C++) Level: 180412 ID: _cDgp0De0EeiRmNzegUH2Xw
   C/C++ Front End Version: 16.1.0(C/C++) Level: 180412 ID: _d-hUqDz8EeiRm9zegUH2Xw
   High-Level Optimizer Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level: 180412 ID: _rAOlVzzZEeiRm9zegUH2Xw
   Low-Level Optimizer Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level: 180412 ID: _ayqxljzZEeiRm9zegUH2Xw
   Intermediate Language Splitter Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _XXdUwfgWEeed2OHRoHeItA
   W-Code to LLVM-IR Translator: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _20SS0Bc1Eeid4OHRoHeItA
   NVVM-IR to PTX Translator: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _ePggcSCAEeiwhrbSB1AWhQ
   /gpfs/software/opt/xlc/16.1/xlC/16.1.0//bin/.orig/xlc: note: XL C/C++ Community Edition is a no-charge product and does not include official IBM support. You can provide feedback at the XL on POWER C/C++ Community Edition forum (http://ibm.biz/xlcpp-linux-ce). For information about a fully supported XL C/C++ compiler, visit XL C/C++ for Linux (http://ibm.biz/xlcpp-linux).

New GCC

I followed along with your compilation instructions and created a dedicated GCC installation in $HOME/TMP/bug-report-temp/usr/local/. Unfortunately, the error stays the same.

# aherten at juron1-adm in ~/TMP/OpenMP-Offloading [15:42:59]
$ /gpfs/software/opt/xlc/16.1/xlC/16.1.0/bin/xlc_configure -o config_xlc -gcc $HOME/TMP/bug-report-temp/usr/local/ -cuda /gpfs/software/opt/cuda/9.2.88 -cuda_cc_major 6 -cuda_cc_minor 0 -nosymlink
INFORMATIONAL: Existing file "config_xlc" has been saved as "config_xlc.2018.7.18.15.43.4".

# aherten at juron1-adm in ~/TMP/OpenMP-Offloading [15:43:04]
$ xlc++ -F config_xlc -qsmp=omp -qoffload pi-gpu.c
In file included from /gpfs/software/opt/cuda/9.2.88/include/fatBinaryCtl.h:19:0,
                 from pi-gpu.fatbin.c:2,
                 from pi-gpu.fatbinwrap.c:2:
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:34:19: error: missing binary operator before token "("
 #if !__has_feature(modules)
                   ^
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:46:42: error: missing binary operator before token "("
 #if !defined(_PTRDIFF_T) || __has_feature(modules)
                                          ^
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:57:39: error: missing binary operator before token "("
 #if !defined(_SIZE_T) || __has_feature(modules)
                                       ^
/gpfs/software/opt/xlc/16.1/xlC/16.1.0/include/stddef.h:71:42: error: missing binary operator before token "("
      !defined(_RSIZE_T)) || __has_feature(modules)
                                          ^
In file included from pi-gpu.fatbin.c:2:0,
                 from pi-gpu.fatbinwrap.c:2:
/gpfs/software/opt/cuda/9.2.88/include/fatBinaryCtl.h:87:62: error: 'size_t' has not been declared
                                                  void* *elf, size_t *esize);
                                                              ^
In file included from pi-gpu.fatbinwrap.c:8:0:
/gpfs/software/opt/cuda/9.2.88/include/crt/host_runtime.h:194:9: error: 'size_t' has not been declared
         size_t size,
         ^
/gpfs/software/opt/cuda/9.2.88/include/crt/host_runtime.h:205:9: error: 'size_t' has not been declared
         size_t size,
         ^
1586-346 (U) An error occurred during code generation.  The code generation return code was 255.

$ xlc -qversion=verbose -F config_xlc
IBM XL C/C++ for Linux, V16.1.0 (Community Edition)
Version: 16.01.0000.0000
Driver Version: 16.1.0(C/C++) Level: 180412 ID: _cDgp0De0EeiRmNzegUH2Xw
C/C++ Front End Version: 16.1.0(C/C++) Level: 180412 ID: _d-hUqDz8EeiRm9zegUH2Xw
High-Level Optimizer Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level: 180412 ID: _rAOlVzzZEeiRm9zegUH2Xw
Low-Level Optimizer Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level: 180412 ID: _ayqxljzZEeiRm9zegUH2Xw
Intermediate Language Splitter Version: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _XXdUwfgWEeed2OHRoHeItA
W-Code to LLVM-IR Translator: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _20SS0Bc1Eeid4OHRoHeItA
NVVM-IR to PTX Translator: 16.1.0(C/C++) and 15.1.7(Fortran) Level 180412 ID: _ePggcSCAEeiwhrbSB1AWhQ
/gpfs/software/opt/xlc/16.1/xlC/16.1.0//bin/.orig/xlc: note: XL C/C++ Community Edition is a no-charge product and does not include official IBM support. You can provide feedback at the XL on POWER C/C++ Community Edition forum (http://ibm.biz/xlcpp-linux-ce). For information about a fully supported XL C/C++ compiler, visit XL C/C++ for Linux (http://ibm.biz/xlcpp-linux).

Do you have any further ideas?

Originally posted by: Rafik_Zurob

Hi Andreas

Assuming pi-gpu.c can be shared, can you please send me the preprocessed output file pi-gpu.ii.c from:

xlc++ -F config_xlc -qsmp=omp -qoffload pi-gpu.c -E > pi-gpu.ii.c

You can either attach the file publicly here (see "Attach a File" below) or privately by adding it to your Files and then sharing it only with me.  (Hover over your name from any of the posts above, click on Files, and then upload the file.  Then share.)

Thanks

Rafik

Originally posted by: Andreas Herten

Hey Rafik,

The file is fine to share. I attached it.

Interesting: For the file I used da xlc_configure file which did not include the -gcc parameter. I did it a second time with the pre-installed GCC 5.4.0 given as -gcc in xlc_configure, but the resulting ii.c file is the same…

-Andreas

Originally posted by: Rafik_Zurob

Hi Andreas.  It occurred to me after I asked for the preprocessed file that the __has_feature(modules) error you're getting is in a preprocessor directive and so the preprocessed file might be truncated.  But the file was correctly preprocessed!  It compiles successfully for me.  This is strange.

I also tried to extract the original pi-gpu.c content from the preprocessed file and to preprocess the file myself with my gcc and compiler.  I got an equivalent preprocessed file to yours.

One thing I noticed is that the compiler you have on gpfs (/gpfs/software/opt/xlc/16.1/xlC/16.1.0/bin/) does not follow the naming conventions for the base directory of the compiler.  i.e. I would have expected it to be (/gpfs/software/opt/xlc/16.1/opt/ibm/xlC/16.1.0/bin/).  That might mean it was not installed using our installer or using rpm.  So just to eliminate the compiler installation as a suspect, can you please try the following script which will download the community edition and install it to your current directory?

The script will wget the community edition from developerWorks (https://www.ibm.com/developerworks/downloads/r/xlcpluslinux/index.html).  If that's not allowed by your organization or if your machine is not connected to the internet, please follow up with your machine admin to get the rpms and adjust the script accordingly.  The rpm files can be downloaded manually from that page by clicking on the "public apt-get, yum, and zypper repositories" link and then clicking on rhel7.

To try it:

mkdir ce
cd ce
./install_xlc_ce_to_curr_dir.sh

[Press 
1 then 
Enter to accept the community edition license]
cd ..

The script will configure the community edition to use gcc from /gpfs/software/opt/gcc/5.4.0/ and CUDA from /gpfs/software/opt/cuda/9.2.88.  If you'd like them from somewhere else, please edit the script accordingly.

Then to verify it works:

./ce/opt/ibm/xlC/16.1.0/bin/xlc++ -qsmp=omp -qoffload pi-gpu.c

If it works, that means the issue is with the community edition installation on /gpfs/software/opt/xlc/16.1/xlC/16.1.0/bin.

Thanks

Rafik

Originally posted by: Andreas Herten

Dear Rafik,

Thank you for the great installation script! Sorry it took me some time to respond.

First: I did the same steps you suggested in my original xlc installation. Downloaded the ZIP with the rpms (instead of them individually), used rpm2cpio | cpio -idmv to extract them, and them moved everything to a separate location. The strange folder structure resulted from the way our software modules are setup on the machine. I then went ahead and changed the TARGETDIR variable in the xlc script.

Now, to start fresh again, I used /opt/ibm as an install location (an old xlC 14.1.0 lives there already). I unpacked the rpms, cpio'd them, and then used rsync to copy them to /opt/ibm/. No need to change the TARGETDIR then.

Unfortunately, this still leads to the original __has_feature() error (I was expecting it, because the error is the same for the old xlC 14.1.0 at the same location). I created custom xlc configs in the dir of pi-gpu.c, once with only CUDA and once with GCC and CUDA. That didn't work. Then I created default (/global) xlc configs, as you suggested. Both led to the same old error.

Do you have any further ideas? (If you want, we can provide access to the machine for you.)

I attached the pi-gpu.c file.

Originally posted by: Rafik_Zurob

Hi Andreas

Yes, I think access to the system is probably the best way to debug at this point.  I'm out of the office from Aug 13-27, so I'll do what I can this week and then hand things over to a colleague to continue the investigation.  If you have the access information available this week, you can send it to rzurob (at ca.ibm.com).

Thanks

Rafik

Originally posted by: Andreas Herten

Hey Rafik,

I'm out of office for the exact same time! I'd suggest we pick things up after vacations? I'll email you concerning all the rest.

(For those of you reading this thread as well or at a later point in time: If you are in need of an update of things, let us know.)

-Andreas

Originally posted by: Andreas Herten

Hey all (lurking or from the future),

Together with Rafik we could determine the problem.

The XL C Compiler was installed correctly, the issue was in our modulefile which set the CPATH environment to point to the include directory of the XL Compiler. This lead to a XL-invoked GCC picking up the wrong headers at some point down the line. The correct way to do things is to not set CPATH for the XL Compiler and rather let the wrappers of XL take care of this.
IBM is currently implementing changes to mitigate this behaviour.

TL;DR: Don't set CPATH for XL Compiler 16.1.0 and OpenMP GPU Offloading should work.

Thank you for helping out!