Originally posted by: Itaru Kitayama

[ 34%] Built target randomtest
[ 34%] Building CXX object nestkernel/CMakeFiles/nestkernel.dir/conn_builder.cpp.o
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/numericdatum.h", line 88: 1586-367 (I) "_ZN12NumericDatumIlXadL_ZN14SLIInterpreter11IntegertypeEEEE6memoryE" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/numericdatum.h", line 88: 1586-367 (I) "_ZN12NumericDatumIdXadL_ZN14SLIInterpreter10DoubletypeEEEE6memoryE" referenced global symbol not marked declare target.
Unexpected empty wcode stack processing Rstr
partitioner: ./util/wcode-builder-body.cpp:1914: virtual bool wc::WCodeBuilder::Rstr(): Assertion `false' failed.
1501-230 (S) Internal compiler error; please contact your Service Representative. For more information visit:
http://www.ibm.com/support/docview.wss?uid=swg21110810
1586-346 (U) An error occurred during code generation.  The code generation return code was 255.
make[2]: *** [nestkernel/CMakeFiles/nestkernel.dir/conn_builder.cpp.o] Error 1
make[1]: *** [nestkernel/CMakeFiles/nestkernel.dir/all] Error 2
make: *** [all] Error 2

Steps to reproduce the error on Minsky:

$ git clone https://www.github.com/ikitayama/nest-simulator

$ make -DCMAKE_TOOLCHAIN_FILE=Platform/JURON_XLC -DCMAKE_INSTALL_PREFIX:PATH=$WORK/opt/5g-xl-16 -Dwith-gsl=ON -Dwith-python=OFF ~/projects/nest-simulator

Originally posted by: Rafik_Zurob

Hello.  I'm unable to reproduce the error using the steps above.  I get a cmake error about Platform/JURON_XLC not being found.

Originally posted by: Itaru Kitayama

Hi - sorry, would you check out the miniNEST-instrumented branch instead, and re-run the cmake command?

Originally posted by: Rafik_Zurob

I'm still getting the same error about Platform/JURON_XLC not being found.  I'm on the miniNEST-instrumented branch:

$ git status
# On branch miniNEST-instrumented
nothing to commit, working directory clean
$

I tried building with -DCMAKE_C_COMPILER="xlc" -DCMAKE_CXX_COMPILER="xlC" -DOpenMP_C_FLAGS="-qsmp=omp -qoffload" -DOpenMP_CXX_FLAGS="-qsmp=omp -qoffload" and that was successful, but I don't think the code had OpenMP offloading code in it./p>

Originally posted by: Itaru Kitayama

Would you apply the patch below to the code base?

diff --git a/nestkernel/conn_builder.cpp b/nestkernel/conn_builder.cpp
index 5753c64..0e9caf0 100644
--- a/nestkernel/conn_builder.cpp
+++ b/nestkernel/conn_builder.cpp
@@ -960,6 +960,7 @@ nest::AllToAllBuilder::connect_()
   }
 }

+#pragma omp declare target
 void
 nest::AllToAllBuilder::inner_connect_( const int tid,
   librandom::RngPtr& rng,
@@ -995,7 +996,7 @@ nest::AllToAllBuilder::inner_connect_( const int tid,
     single_connect_( *sgid, *target, target_thread, rng );
   }
 }
-
+#pragma omp end declare target
 /**
  * Solves the connection of two nodes on a AllToAll basis with
  * structural plasticity. This means this method is used by the

Originally posted by: w2yehia

Hi Itaru, I will try to help you out.

I'm having trouble following your reproduce instructions; i'm hitting the same issues as Rafik.

I will continue trying but in the meantime:

Since what you are having is a compile time problem, it would be easiest if you can reproduce for us the preprocessed file and give us the full compiler invocation command.

One way to get the compiler invocation command is to run the "make" step with "VERBOSE=1" (e.g. $ make VERBOSE=1 mytarget), and file the compile step right before you get the error. Do not use the -j option.
You should be able to "cd" into whatever directory that "make" is cd'ing, run the compile step that you obtained earlier, and be able to reproduce the same error.
To obtain the preprocessed file, remove any "-o" option from the compiler command, and append the "-E" option. That will ask the compiler to emit the preprocessed output to stdout.
You can then attach the preprocessed file here, and paste the compiler invocation command.

Originally posted by: w2yehia

I was able to reproduce the error after marking nest::AllToAllBuilder::inner_connect_ declare target.

Originally posted by: Itaru Kitayama

Another similar error:

[...]

[ 34%] Built target randomtest
[ 34%] Building CXX object nestkernel/CMakeFiles/nestkernel.dir/conn_builder.cpp.o
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.cpp", line 352: 1586-367 (I) "_ZTI14NotImplemented" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.cpp", line 360: 1586-367 (I) "_ZTI14NotImplemented" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.cpp", line 398: 1586-367 (I) "_ZTI22WrappedThreadException" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.cpp", line 423: 1586-367 (I) "_ZTI22WrappedThreadException" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/lockptrdatum_impl.h", line 41: 1586-367 (I) "_ZTI5Datum" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.h", line 135: 1586-367 (I) "_ZTI14NotImplemented" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.h", line 141: 1586-367 (I) "_ZTI14NotImplemented" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/nestkernel/conn_builder.h", line 146: 1586-367 (I) "_ZTI14NotImplemented" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/genericdatum.h", line 115: 1586-367 (I) "_ZTI5Datum" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/numericdatum.h", line 88: 1586-367 (I) "_ZN12NumericDatumIlXadL_ZN14SLIInterpreter11IntegertypeEEEE6memoryE" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/genericdatum.h", line 115: 1586-367 (I) "_ZTI5Datum" referenced global symbol not marked declare target.
"/gpfs/homeb/pcp0/pcp0151/projects/nest-simulator/sli/numericdatum.h", line 88: 1586-367 (I) "_ZN12NumericDatumIdXadL_ZN14SLIInterpreter10DoubletypeEEEE6memoryE" referenced global symbol not marked declare target.
Unexpected empty wcode stack processing Rstr
partitioner: ./util/wcode-builder-body.cpp:1914: virtual bool wc::WCodeBuilder::Rstr(): Assertion `false' failed.
1501-230 (S) Internal compiler error; please contact your Service Representative. For more information visit:
http://www.ibm.com/support/docview.wss?uid=swg21110810
1586-346 (U) An error occurred during code generation.  The code generation return code was 255.
make[2]: *** [nestkernel/CMakeFiles/nestkernel.dir/conn_builder.cpp.o] Error 1
make[1]: *** [nestkernel/CMakeFiles/nestkernel.dir/all] Error 2
make: *** [all] Error 2

Originally posted by: Itaru Kitayama

You can check out the gpus branch from https://github.com/ikitayama/nest-simulator

Originally posted by: w2yehia

Hi Itaru, the issue you were seeing when compiling conn_builder.cpp has been fixed. 
The fix will be available in a future version of the compiler (no particular dates are known at the moment).

Originally posted by: Itaru Kitayama

I appreciate it if you could tell us about the specific version info. Thanks!

Originally posted by: Rafik_Zurob

Hello Kitayama-san

It was too late to include the fix in the GA (first) version of XL C/C++ for Linux, V16.1.1.  We are therefore targeting the fix for a PTF of XL C/C++ for Linux, V16.1.1.  We're still working on determining the best date for the PTF, and are taking this issue into account.

Note that we released the community (free of charge) edition of XL C/C++ for Linux, V16.1.1 earlier this week.  The regular (license charge) edition which comes with access to IBM support is planned for release later this year.  If you need the fix right away and you have an active support subscription to any version of XL C/C++ for Linux, please contact IBM support and ask them for an interim fix for XL Compilers RTC defect 158349.

I'm sorry for any inconvenience this delay may cause.

Thanks!

Rafik

Originally posted by: Itaru Kitayama

Hello Rafik,

Has the version which has the bug fix been released  as something like V16.1.2 or the like as the community

edition? If so, I'll let our sysadmin (Andreas) know so he can install the right one.

Thanks!

Itaru.

Originally posted by: Rafik_Zurob

Hi Itaru

We expect this fix to be available in the next PTF and community edition refresh, which is currently planned for February.

Cheers
 

Rafik

Originally posted by: Itaru Kitayama

Has the Community Edition release date been fixed? Thanks!

Originally posted by: w2yehia

Hi Itaru,

I forgot to mention one thing. In your application, you might need to replace all uses of

  << std::endl

in any device code (or code reachable from a device function, e.g. foo is declare target that calls a non-declare target bar) with

  << "\n"

because std::endl is not supported on the device. Please keep that in mind when you use the new version of the compiler in February.

Originally posted by: Itaru Kitayama

Is this the version we talked about?

https://www-01.ibm.com/support/docview.wss?uid=ibm10870440

Originally posted by: Rafik_Zurob

Hi Itaru.  Yes, this is the right PTF.  I'm not sure if you have access to the download.  If you do, you can install the PTF over the community edition to update the community edition.
To enable the fix for the internal compiler error described in this thread, please specify -qxflag=deep_promote on the compile step.  Don't forget to also remove std::endl from device code as described above.
Thanks!

Originally posted by: Itaru Kitayama

Hi Rafik,

Thanks for your confirmation. Do you support static linking with this PTF?

Originally posted by: Rafik_Zurob

Hi Itaru.  Depends on what you want to statically link.  If you want to statically link the XL libraries (including the SMP / OpenMP library), you can link with "-qstaticlink=xllibs".  But if you want to statically link the NVIDIA CUDA libraries (libcuda and libcudart), we do not support that.

Originally posted by: Itaru Kitayama

Hi Rafik - should the -qxflag=deep_promote flag carried over to the future XL releases, be it 16.1.1.3 or 17.x? 

Originally posted by: Rafik_Zurob

Hi Itaru.  Yes, you can expect this -qxflag to continue to exist or to become default behaviour.