After adding 3 GB space to paging space, I have successfully installed the Xgboost (v1.7.5) finally. Thanks for your help.
Original Message:
Sent: Fri September 29, 2023 01:44 AM
From: Ayappan P
Subject: Error during installing xgboost
Try increasing below ulimit values.
ulimit -d unlimited
ulimit -m unlimited
ulimit -n 20000
------------------------------
Ayappan P
Original Message:
Sent: Thu September 28, 2023 05:21 PM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
I just tried installing the xgboost using a different filesystem's storage for extracting the source code. I got exact the same result: the building process failed when building CXX objects reached 96 %. SO this is not related to the filesystem holding the building contents. It must be something else.
Thanks
Mi
------------------------------
Mi Wang
Original Message:
Sent: Thu September 28, 2023 11:45 AM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
That filesystem (for extracting the source code) has 74 GB free space. All other system related filesytems' space usage doesn't exceed 65 % except /usr, which has > 1GB free space. I am wondering could there be other type storage causing the error (I checked swap space and it's also fine)?
Thanks
Mi
------------------------------
Mi Wang
Original Message:
Sent: Thu September 28, 2023 03:11 AM
From: Ranjit Ranjan
Subject: Error during installing xgboost
Hi,
is the disk partition under which you extracted the source code , having the enough disk space ?
check this also.
once you hit this error, capture 'df -k' also.
Thanks
Ranjit
------------------------------
Ranjit Ranjan
Original Message:
Sent: Wed September 27, 2023 05:15 PM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
I have followed your recommendation and the installation has got into normal 'building CXX...' stage. Then it hit with the error (see below for error messages) related to space. I have tried to increase the space for / and /opt filesystems (the /tmp has about 10 GB free space), but still got the same error. Can you please help to find out where this 'not enough space' complaint is for?
Thanks
Mi
......
[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o
[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o
/bin/sh[3]: 8389106 Killed
/bin/sh[3]: 8126970 Killed
/bin/sh[3]: 8782240 Killed
/bin/sh[3]: 8454498 Killed
/bin/sh[3]: 9109762 Killed
/bin/sh[3]: 10813776 Killed
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make: The signal code from the last command is 9.
Stop.
make: The error code from the last command is 2.
......
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
/bin/sh[3]: 11206954 Killed
error: [Errno 12] Not enough space
/bin/sh[3]: 11862358 Killed
/bin/sh[3]: 13435344 Killed
------------------------------
Mi Wang
Original Message:
Sent: Wed September 27, 2023 02:36 AM
From: Ranjit Ranjan
Subject: Error during installing xgboost
Hi Wang,
Looks , LIBPATH issue.
are you able to run 'cmake' from the shell session? just type 'cmake' and enter'.
are you setting any LIBPATH in your build session ?
Try without setting any LIBPATH.
if you still see the above error message 'Could not load program cmake', then collect the output of below commands:
- rpm -qa | grep -i cmake
- ldd /opt/freeware/bin/cmake
Thanks
Ranjit
------------------------------
Ranjit Ranjan
Original Message:
Sent: Tue September 26, 2023 02:44 PM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
After I mode the modification to the socket.h file, I am now getting below new error message. Please take a look.
Thanks
Mi
INFO:XGBoost build_ext:Run CMake command: ['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=1', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']
Could not load program cmake:
Symbol resolution failed for cmake because:
Symbol _ZTINSt6thread6_StateE (number 351) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZTVNSt6thread6_StateE (number 352) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt6thread4joinEv (number 381) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt18condition_variableD1Ev (number 403) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt18condition_variableC1Ev (number 404) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (number 448) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt18condition_variable10notify_oneEv (number 449) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt18condition_variable10notify_allEv (number 450) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt6thread6_StateD2Ev (number 461) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt6thread15_M_start_threadESt10unique_ptrINS_6_StateESt14default_deleteIS1_EEPFvvE (number 462) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Symbol _ZNSt6thread20hardware_concurrencyEv (number 463) is not exported from dependent
module /opt/freeware/lib/libstdc++.a[libstdc++.so.6].
Examine .loader section symbols with the 'dump -Tv' command.
------------------------------
Mi Wang
Original Message:
Sent: Tue September 26, 2023 12:24 AM
From: Ranjit Ranjan
Subject: Error during installing xgboost
For 1.7.5, you also need to apply the patch for socket.h . I think, you know the changes as you mentioned in the starting post message.
Apply that change and try it out.
------------------------------
Ranjit Ranjan
Original Message:
Sent: Mon September 25, 2023 05:21 PM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
I just tried he installation of xgboost-1.7.5 on our Python 3.9 environment. I did the same environment variable exports. Please see following modifications to the 3 py scripts. The installation failed (see below). Please help.
Thanks
Mi
Modification:
/tmp/xgboost/xgboost-1.7.5: diff setup.py_orig setup.py
46a47,48
> elif system() == 'AIX':
> name = 'libxgboost.a'
/tmp/xgboost/xgboost-1.7.5/xgboost: diff libpath.py_orig libpath.py
51a52,53
> elif sys.platform.startswith('aix'):
> dll_path = [os.path.join(p, 'libxgboost.a') for p in dll_path]
/tmp/xgboost/xgboost-1.7.5/xgboost: diff core.py_orig core.py
206c206
< lib = ctypes.cdll.LoadLibrary(lib_path)
---
> lib = ctypes.cdll.LoadLibrary(lib_path + '(libxgboost.so)')
Error message:
[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o
In file included from /usr/opt/sas/python3a/xgboost-1.7.5/build/temp.aix-7205-2319-64-cpython-39/xgboost/include/xgboost/collective/socket.h:55,
from /usr/opt/sas/python3a/xgboost-1.7.5/build/temp.aix-7205-2319-64-cpython-39/xgboost/src/collective/socket.cc:4:
/usr/opt/sas/python3a/xgboost-1.7.5/build/temp.aix-7205-2319-64-cpython-39/xgboost/include/xgboost/collective/socket.h: In member function 'xgboost::collective::SockDomain xgboost::collective::TCPSocket::Domain() const':
/usr/opt/sas/python3a/xgboost-1.7.5/build/temp.aix-7205-2319-64-cpython-39/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?
294 | getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);
| ^~~~~~~~~
/usr/opt/sas/python3a/xgboost-1.7.5/build/temp.aix-7205-2319-64-cpython-39/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'
73 | #define XGBOOST_EXPECT(cond, ret) __builtin_expect((cond), (ret))
| ^~~~
/usr/opt/sas/python3a/xgboost-1.7.5/build/temp.aix-7205-2319-64-cpython-39/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'
293 | xgboost_CHECK_SYS_CALL(
| ^~~~~~~~~~~~~~~~~~~~~~
------------------------------
Mi Wang
Original Message:
Sent: Mon September 25, 2023 01:56 PM
From: Ranjit Ranjan
Subject: Error during installing xgboost
Above export env is okay. No need to set any LIBPATH.
I don't have python3.7 env to try it out. But it should work.
I Don't see any issue in python3.9.
Try freshly from start ,by extracting the source code of 1.6.2 , applying the suggested patches and run the same command for installation.
you can add -v for verbose output 'python3 -m pip install . --root / -v'
Thanks
Ranjit
------------------------------
Ranjit Ranjan
Original Message:
Sent: Mon September 25, 2023 10:35 AM
From: Mi Wang
Subject: Error during installing xgboost
By the way, I used the following command to install the package:
python3 -m pip install . --root /
Thanks
Mi
------------------------------
Mi Wang
Original Message:
Sent: Mon September 25, 2023 10:08 AM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
I use the following for defining env parameters. Should I also add export LIBPATH=/opt/freeware/lib64:$LIBPATH as well?
export OBJECT_MODE=64
export CC="gcc -maix64 -pthread"
export CXX="g++ -maix64 -pthread"
Thanks
Mi
------------------------------
Mi Wang
Original Message:
Sent: Mon September 25, 2023 02:48 AM
From: Ranjit Ranjan
Subject: Error during installing xgboost
what are the env exports used ?
cmake is not able to find pthread.h that's why it's throwing above error.
Thanks
Ranjit
------------------------------
Ranjit Ranjan
Original Message:
Sent: Fri September 22, 2023 02:55 PM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
As our Python 3.9 environment is temporarily not available for me, I have tried installing xgboost 1.6.2 on another Python 3.7 environment. I used the same steps from you and got the following error. Can you please help?
Thanks
Mi
-- xgboost VERSION: 1.6.2
-- Setting build type to 'Release' as none was specified.
-- Looking for pthread.h
-- Looking for pthread.h - not found
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
/opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
/opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:151 (find_package)
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-req-build-4u3rb93v/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-req-build-4u3rb93v/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
WARNING:XGBoost build_ext:Disabling OpenMP support.
INFO:XGBoost build_ext:Run CMake command: ['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']
-- CMake version 3.22.0
-- xgboost VERSION: 1.6.2
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
------------------------------
Mi Wang
Original Message:
Sent: Wed September 20, 2023 08:37 AM
From: Ranjit Ranjan
Subject: Error during installing xgboost
Okay, Wang.
Make the changes for below file also . you can take the change from previous thread.
- setup.py,
- xgboost/libpath.py,
- xgboost/core.py
To build and install, run the command 'python3 -m pip install . --root /usr/local'.
Before using it, export below two path.
export LIBPATH=/usr/local/opt/freeware/lib64/python3.9/site-packages/xgboost/lib
export PYTHONPATH=/usr/local/opt/freeware/lib64/python3.9/site-packages/
This should work and it worked for me. :)
------------------------------
Ranjit Ranjan
Original Message:
Sent: Tue September 19, 2023 10:51 AM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjit,
Our AIX server / other machine don't have git installed. I am wondering if I can do the following instead:
1. download the Xgboost package from PYPI site, upload it into a location on our AIX server, e.g.: /tmp/test and ungzip / untar the source code (folders / files) there. A subfolder called 'xgboost-1.7.5' will be created there.
2. Under the /tmp/test/xgboost-1.7.5 folder, continue the following steps:
mkdir build
cd build
export CC="gcc -maix64 -pthread"
export CXX="g++ -maix64 -pthread"
export OBJECT_MODE=64
............
Thanks
Mi
------------------------------
Mi Wang
Original Message:
Sent: Tue September 19, 2023 01:29 AM
From: Ranjit Ranjan
Subject: Error during installing xgboost
Hi Wang,
This is also discussed in previous thread https://community.ibm.com/community/user/power/discussion/issue-with-c-c-compiling-during-the-pip-installation-for-xgboost.
Could you try the steps provided by Ayappan and let us know if it's working or not.
Thanks
------------------------------
Ranjit Ranjan
Original Message:
Sent: Mon September 18, 2023 10:22 AM
From: Mi Wang
Subject: Error during installing xgboost
Hi Ranjan,
This is actually the continued discussion regarding issues with xgboost package installation (re: previous thread "Issue with C / C++ Compiling during the PIP installation for Xgboost"). We have now upgraded to Python 3.9 eco system. I have tried the installation of xgboost 1.7.5 the way you did the installation: 1) set the environment variables OBJECT_MODE, CC and CXX; 2) modified the include/xgboost/collective/socket.h file, replacing "#ifndef __PASE__" with "#if defined(__PASE__) && !defined(AIX)". But I got the following error message. Can you please help?
Thanks
Mi
..........
File "/opt/freeware/lib64/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/opt/freeware/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
self._build_extensions_serial()
File "/opt/freeware/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
self.build_extension(ext)
File "/tmp/pip-install-m1nm1aqx/xgboost_fde5d35a81024fff9cce8bb9ca0d86c5/setup.py", line 206, in build_extension
self.build_cmake_extension()
File "/tmp/pip-install-m1nm1aqx/xgboost_fde5d35a81024fff9cce8bb9ca0d86c5/setup.py", line 154, in build_cmake_extension
os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
File "/tmp/pip-install-m1nm1aqx/xgboost_fde5d35a81024fff9cce8bb9ca0d86c5/setup.py", line 49, in lib_name
return name
UnboundLocalError: local variable 'name' referenced before assignment
[end of output]
------------------------------
Mi Wang
------------------------------