Originally posted by: Teej42

This is a followup on the prior page regarding installation of SciPy on AIX.

Based on my work with NumPy - https://github.com/xianyi/OpenBLAS/issues/1803 - I set up the following commands to run:

# export LDFLAGS='-lpthread'
# export CC='gcc -pthread -lm -lgcc -lgomp -lgfortran -maix64'
# export PATH=/opt/freeware/bin:$PATH
# python_64 setup.py install > install.txt 2>&1

There are a number of conflict errors coming from the codebase:  

In file included from /opt/freeware/include/python2.7/Python.h:44:0,
                 from scipy/special/cephes/mconf.h:61,
                 from scipy/special/cephes/chbevl.c:61:
/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/include-fixed/unistd.h:210:16: error: conflicting types for 'lseek64'
 extern off64_t lseek64(int, off64_t, int);
                ^~~~~~~
/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/include-fixed/unistd.h:208:14: note: previous declaration of 'lseek64' was here
 extern off_t lseek(int, off_t, int);
              ^

This occurred several times with other functions, such as ftruncate64, lockf64, et cetera.  Then I have this:

error: Command "gcc -pthread -lm -lgcc -lgomp -lgfortran -maix64 -fno-strict-aliasing -D_GNU_SOURCE -fPIC -fno-strict-aliasing -fwrapv -D_LINUX_SOURCE_COMPAT -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DNDEBUG -D_GNU_SOURCE -fPIC -fno-strict-aliasing -fwrapv -I/medstat/advantage/scratch/focal_point/download/scipy-1.1.0/scipy/special -I/opt/freeware/lib64/python2.7/site-packages/numpy-1.15.3-py2.7-aix-7.1.egg/numpy/core/include -I/opt/freeware/include/python2.7 -I/medstat/advantage/scratch/focal_point/download/scipy-1.1.0/scipy/special/c_misc -Iscipy/_lib -I/opt/freeware/lib64/python2.7/site-packages/numpy-1.15.3-py2.7-aix-7.1.egg/numpy/core/include -I/opt/freeware/include/python2.7 -c scipy/special/cephes/chbevl.c -o build/temp.aix-7.1-2.7/scipy/special/cephes/chbevl.o -MMD -MF build/temp.aix-7.1-2.7/scipy/special/cephes/chbevl.o.d" failed with exit status 1

Please advise on the resolution for this issue.  Thanks!

Originally posted by: Teej42

Pandas package is experiencing the same issue.  

So need to find a solution to solve for both packages.

Note, ran setting as:

# export LDFLAGS='-lpthread'
# export CC='gcc -lpthread -lm -lgcc -lgomp -lgfortran -maix64'
# export PATH=/opt/freeware/bin:$PATH
# python_64 setup.py install > install.txt 2>&1

Had to modify:  pandas/_libs/window.cpp:11037

from:

  __pyx_t_1 = (std::isnan(__pyx_v_val) != 0);

To: 

  __pyx_t_1 = (std::isnan((long double)__pyx_v_val) != 0);

Any advice?

Attachment(s)

install-pandas.txt   28 KB 1 version

Originally posted by: AyappanP

In this file "/opt/freeware/include/python2.7/pyconfig-ppc64.h" remove the below line and try again.

#define _LARGE_FILES 1

Originally posted by: Teej42

That resolves the initial issue for both.  New issue for SciPy - despite including -lgomp in CC, it's throwing missing omp links - 

C compiler: gcc -pthread -lm -lgcc -lgomp -lgfortran -maix64 -fno-strict-aliasing -D_GNU_SOURCE -fPIC -fno-strict-aliasing -fwrapv -D_LINUX_SOURCE_COMPAT -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DNDEBUG -D_GNU_SOURCE -fPIC -fno-strict-aliasing -fwrapv

compile options: '-DHAVE_CBLAS -I/opt/freeware/lib64/python2.7/site-packages/numpy-1.15.3-py2.7-aix-7.1.egg/numpy/core/include -I/opt/freeware/lib64/python2.7/site-packages/numpy-1.15.3-py2.7-aix-7.1.egg/numpy/core/include -I/opt/freeware/include/python2.7 -c'
/opt/freeware/lib64/python2.7/config/ld_so_aix /usr/bin/gcc -maix64 -pthread -bI:/opt/freeware/lib64/python2.7/config/python.exp -L. -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib/lib64 -L/usr/lib -Wl,-blibpath:/opt/freeware/lib64:/opt/freeware/lib:/usr/lib:/lib -Wl,-brtl build/temp.aix-7.1-2.7/scipy/cluster/_vq.o -L/opt/OpenBLAS/lib -L/opt/freeware/lib64 -Lbuild/temp.aix-7.1-2.7 -Wl,-R/opt/OpenBLAS/lib -lopenblas -lopenblas -o build/lib.aix-7.1-2.7/scipy/cluster/_vq.so
[...]

ld: 0711-317 ERROR: Undefined symbol: .omp_get_num_threads
ld: 0711-317 ERROR: Undefined symbol: .omp_get_thread_num
ld: 0711-317 ERROR: Undefined symbol: .omp_set_num_threads
ld: 0711-317 ERROR: Undefined symbol: .GOMP_parallel
ld: 0711-317 ERROR: Undefined symbol: .omp_in_parallel
ld: 0711-317 ERROR: Undefined symbol: .omp_get_max_threads

Right now I'm trying to remember which env var to include -lgomp on.  If you can quickly let me know on Slack, would be appreciated.

Originally posted by: Teej42

Okay, I did try LDFLAGS, and it broke in a weird way.  I will need to wipe everything and restore everything step by step, and see if it would resolve the issues.  Logs attached if you are curious.

Attachment(s)

install-scipy.txt   348 KB 1 version

Originally posted by: Teej42

Been trying a few things, and this is where I am right now:

# export CFLAGS='-lpthread -lm -lgcc -lgomp -lgfortran -maix64'
# export LDFLAGS='-lpthread -lm -lgcc -lgomp -lgfortran -maix64'
# export FFLAGS='-lgomp -maix64'
# export CC='gcc -lpthread -lm -lgcc -lgomp -lgfortran -maix64'
# export PATH=/opt/freeware/bin:$PATH
# python_64 setup.py install > install-scipy.txt 2>&1

See attached for the log (this was the second time it ran, but same stopping point.  If I manually remove some settings, it would link successfully, but I don't know what after that (probably installation).

Help, pretty please?  :-)

Attachment(s)

install-scipy.txt   20 KB 1 version

Originally posted by: AyappanP

Creating of library failed due to improper options passed to AIX linker

"-Wl,-rpath="/opt/OpenBLAS/lib" " AIX linker will not recognize this "-rpath". 

Originally posted by: Teej42

Please review results of my research on https://github.com/scipy/scipy/issues/9452 for this issue.  It would help a lot if we improve SciPy's behavior to better work with AIX. Thanks.

Originally posted by: AyappanP

Just checked the issue. I am wondering why we didn't hit the "rpath" issue with 32bit scipy build with netlib blas library ?

I think the best way to fix this is to add "aix" check as well.

sys.platform == 'darwin' or sys.platform[:3] == 'aix'

Originally posted by: Teej42

I know that for that function on 32 bit, we have:

/opt/freeware/lib/python2.7/site-packages/distutils/fcompiler/gnu.py

While on 64 bit, it's:

/opt/freeware/lib64/python2.7/site-packages/numpy-1.15.3-py2.7-aix-7.1.egg/numpy/distutils/fcompiler/gnu.py

Not sure if NumPy overrode the 32 bit's logic, but the function appears to be the same.  Very good question here.  Maybe it's due to the fact that when I used 32 bit, pip install worked perfectly, while 64 bit installation required a few changes.

If I find the time, I will try to figure it out.

Originally posted by: sanket

Here is the tutorial how to install ML tools (NumPy, SciPy, Matplotlib, Pandas, scikit-learn and other on AIX)

https://developer.ibm.com/tutorials/machine-learning-with-python-on-aix/

I hope this will be helpful.