AIX Open Source

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Wang,
Looks like something is wrong with your setuptools. 
You can remove the existing setuptools and install it freshly. 
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools*dist-info

python3 -m ensurepip
python3 -m pip install --upgrade setuptools

Then you can try installing xgboost. 

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

Hi Ranjit / Ayappan,

After I reinstalled the default setuptools (47.1.0), now I got the similar error when installing xgboost as you did. Please help to resolve the issue:

  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 203, in build_extension
        self.build_cmake_extension()
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 153, in build_cmake_extension
        os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 48, in lib_name
        return name
    UnboundLocalError: local variable 'name' referenced before assignment

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Fri May 26, 2023 07:40 AM
From: Ayappan P
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,
Looks like something is wrong with your setuptools. 
You can remove the existing setuptools and install it freshly. 
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools*dist-info

python3 -m ensurepip
python3 -m pip install --upgrade setuptools

Then you can try installing xgboost. 

------------------------------
Ayappan P

Original Message:
Sent: Wed May 24, 2023 05:22 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Wed May 24, 2023 10:48 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Tue May 23, 2023 02:58 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

------------------------------
Mi Wang

Original Message:
Sent: Mon May 22, 2023 03:27 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Thu May 18, 2023 02:50 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Thu May 18, 2023 12:57 PM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Wed May 17, 2023 12:16 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Tue May 16, 2023 02:28 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Mon May 15, 2023 02:37 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

------------------------------
Mi Wang
------------------------------

Hi Wang,

So direct pip installation will not work. We need to make some source code change to make it compatible with AIX.  
Try below steps and see if it's working for you. 

First download the source from below link
https://files.pythonhosted.org/packages/25/e0/f00261d3357eafbe7c657de718abd3c0f6613633a38d05350f63ee099cde/xgboost-1.6.2.tar.gz

1. Extract it : tar -xvf xgboost-1.6.2.tar.gz
2. cd xgboost-1.6.2
3. Export the env which I had shared previously.
4. Run 'python3.7 setup.py install'. Same below error will be reported.

    self.build_cmake_extension()

  File "setup.py", line 153, in build_cmake_extension

    os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))

  File "setup.py", line 48, in lib_name

    return name

UnboundLocalError: local variable 'name' referenced before assignment.

To fix this , we need to modify the source as explained. Make sure to use proper spacing for modifying below source file as these are python code and spacing matters. Use 4 spacing for modifying the code. 

5. Take a backup of setup.py and modify setup.py with below changes.

# diff -u setup.py.orig setup.py

--- setup.py.orig       2023-05-29 01:40:36.295265022 +0000

+++ setup.py    2023-05-29 01:43:15.045066128 +0000

@@ -45,6 +45,8 @@

         name = 'libxgboost.dylib'

     elif system() == 'Windows':

         name = 'xgboost.dll'

+    elif system() == 'AIX':

+        name = 'libxgboost.a'

     return name

6. Take a backup of xgboost/libpath.py  and modify  xgboost/libpath.py with below change

(0) root @ pokndd3: /ranjit/xgboost-1.6.2/xgboost

# diff -u libpath.py.orig libpath.py

--- libpath.py.orig     2023-05-29 01:57:12.505062007 +0000

+++ libpath.py  2023-05-29 02:16:33.611085204 +0000

@@ -50,6 +50,8 @@

         dll_path = [os.path.join(p, 'libxgboost.dylib') for p in dll_path]

     elif sys.platform == 'cygwin':

         dll_path = [os.path.join(p, 'cygxgboost.dll') for p in dll_path]

+    elif sys.platform.startswith('aix'):

+        dll_path = [os.path.join(p, 'libxgboost.so') for p in dll_path]

     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]

7. python3.7 setup.py install   (This should pass now)
   
   
8. Execute below:

cp /opt/freeware/lib64/python3.7/site-packages/xgboost/lib/libxgboost.a /opt/freeware/lib

cd /opt/freeware/lib/

ar -x libxgboost.a

9. Now, xgboost can be imported.

# python3

Python 3.7.15 (default, Dec 15 2022, 03:08:16)

[GCC 8.3.0] on aix7

Type "help", "copyright", "credits" or "license" for more information.

>>> import xgboost

------------------------------
Ranjit Ranjan

Original Message:
Sent: Fri May 26, 2023 04:10 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit / Ayappan,

After I reinstalled the default setuptools (47.1.0), now I got the similar error when installing xgboost as you did. Please help to resolve the issue:

  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 203, in build_extension
        self.build_cmake_extension()
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 153, in build_cmake_extension
        os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 48, in lib_name
        return name
    UnboundLocalError: local variable 'name' referenced before assignment

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Fri May 26, 2023 07:40 AM
From: Ayappan P
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,
Looks like something is wrong with your setuptools. 
You can remove the existing setuptools and install it freshly. 
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools*dist-info

python3 -m ensurepip
python3 -m pip install --upgrade setuptools

Then you can try installing xgboost. 

------------------------------
Ayappan P

Original Message:
Sent: Wed May 24, 2023 05:22 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Wed May 24, 2023 10:48 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Tue May 23, 2023 02:58 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

------------------------------
Mi Wang

Original Message:
Sent: Mon May 22, 2023 03:27 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Thu May 18, 2023 02:50 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Thu May 18, 2023 12:57 PM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Wed May 17, 2023 12:16 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Tue May 16, 2023 02:28 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Mon May 15, 2023 02:37 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

------------------------------
Mi Wang
------------------------------

Hi Ranjit,

I have followed the steps you suggested. But got following errors when doing step 7: running 'python3.7 setup.py install'. Please help.

Thanks

Mi

-- Check for working CXX compiler: /opt/freeware/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/freeware/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- CMake version 3.22.0
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT - Failed
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT - Success
-- xgboost VERSION: 1.6.2
-- Setting build type to 'Release' as none was specified.
-- Looking for pthread.h
-- Looking for pthread.h - not found
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:151 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
WARNING:XGBoost build_ext:Disabling OpenMP support.
INFO:XGBoost build_ext:Run CMake command: ['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']
-- CMake version 3.22.0
-- xgboost VERSION: 1.6.2
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:151 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 197, in build_cmake_extension
    self.build(src_dir, build_dir, gen, build_tool, use_omp=1)
  File "setup.py", line 134, in build
    subprocess.check_call(cmake_cmd, cwd=build_dir)
  File "/opt/freeware/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=1', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 372, in <module>
    url='https://github.com/dmlc/xgboost')
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 307, in run
    super().run()
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/install.py", line 65, in run
    orig.install.run(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 205, in build_extension
    self.build_cmake_extension()
  File "setup.py", line 200, in build_cmake_extension
    self.build(src_dir, build_dir, gen, build_tool, use_omp=0)
  File "setup.py", line 134, in build
    subprocess.check_call(cmake_cmd, cwd=build_dir)
  File "/opt/freeware/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']' returned non-zero exit status 1.
root [bastdctdv64] /tmp/new-libs-1/xgboost-1.6.2:

Hi Wang,

So direct pip installation will not work. We need to make some source code change to make it compatible with AIX.  
Try below steps and see if it's working for you. 

First download the source from below link
https://files.pythonhosted.org/packages/25/e0/f00261d3357eafbe7c657de718abd3c0f6613633a38d05350f63ee099cde/xgboost-1.6.2.tar.gz

1. Extract it : tar -xvf xgboost-1.6.2.tar.gz
2. cd xgboost-1.6.2
3. Export the env which I had shared previously.
4. Run 'python3.7 setup.py install'. Same below error will be reported.

    self.build_cmake_extension()

  File "setup.py", line 153, in build_cmake_extension

    os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))

  File "setup.py", line 48, in lib_name

    return name

UnboundLocalError: local variable 'name' referenced before assignment.

To fix this , we need to modify the source as explained. Make sure to use proper spacing for modifying below source file as these are python code and spacing matters. Use 4 spacing for modifying the code. 

5. Take a backup of setup.py and modify setup.py with below changes.

# diff -u setup.py.orig setup.py

--- setup.py.orig       2023-05-29 01:40:36.295265022 +0000

+++ setup.py    2023-05-29 01:43:15.045066128 +0000

@@ -45,6 +45,8 @@

         name = 'libxgboost.dylib'

     elif system() == 'Windows':

         name = 'xgboost.dll'

+    elif system() == 'AIX':

+        name = 'libxgboost.a'

     return name

6. Take a backup of xgboost/libpath.py  and modify  xgboost/libpath.py with below change

(0) root @ pokndd3: /ranjit/xgboost-1.6.2/xgboost

# diff -u libpath.py.orig libpath.py

--- libpath.py.orig     2023-05-29 01:57:12.505062007 +0000

+++ libpath.py  2023-05-29 02:16:33.611085204 +0000

@@ -50,6 +50,8 @@

         dll_path = [os.path.join(p, 'libxgboost.dylib') for p in dll_path]

     elif sys.platform == 'cygwin':

         dll_path = [os.path.join(p, 'cygxgboost.dll') for p in dll_path]

+    elif sys.platform.startswith('aix'):

+        dll_path = [os.path.join(p, 'libxgboost.so') for p in dll_path]

     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]

7. python3.7 setup.py install   (This should pass now)
   
   
8. Execute below:

cp /opt/freeware/lib64/python3.7/site-packages/xgboost/lib/libxgboost.a /opt/freeware/lib

cd /opt/freeware/lib/

ar -x libxgboost.a

9. Now, xgboost can be imported.

# python3

Python 3.7.15 (default, Dec 15 2022, 03:08:16)

[GCC 8.3.0] on aix7

Type "help", "copyright", "credits" or "license" for more information.

>>> import xgboost

------------------------------
Ranjit Ranjan

Original Message:
Sent: Fri May 26, 2023 04:10 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit / Ayappan,

After I reinstalled the default setuptools (47.1.0), now I got the similar error when installing xgboost as you did. Please help to resolve the issue:

  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 203, in build_extension
        self.build_cmake_extension()
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 153, in build_cmake_extension
        os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 48, in lib_name
        return name
    UnboundLocalError: local variable 'name' referenced before assignment

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Fri May 26, 2023 07:40 AM
From: Ayappan P
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,
Looks like something is wrong with your setuptools. 
You can remove the existing setuptools and install it freshly. 
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools*dist-info

python3 -m ensurepip
python3 -m pip install --upgrade setuptools

Then you can try installing xgboost. 

------------------------------
Ayappan P

Original Message:
Sent: Wed May 24, 2023 05:22 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Wed May 24, 2023 10:48 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Tue May 23, 2023 02:58 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

------------------------------
Mi Wang

Original Message:
Sent: Mon May 22, 2023 03:27 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Thu May 18, 2023 02:50 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Thu May 18, 2023 12:57 PM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Wed May 17, 2023 12:16 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Tue May 16, 2023 02:28 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Mon May 15, 2023 02:37 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

------------------------------
Mi Wang
------------------------------

Can you try the below steps

git clone https://github.com/dmlc/xgboost
cd xgboost
git checkout v1.6.2
git submodule init
git submodule update
mkdir build
cd build
export CC="gcc -maix64 -pthread"
export CXX="g++ -maix64 -pthread"
export OBJECT_MODE=64
cmake ..

cd ../python-package

Modify below three files

# diff -u setup.py_orig setup.py
--- setup.py_orig       2023-05-30 02:07:56.160027553 +0000
+++ setup.py    2023-05-30 02:19:35.710912296 +0000
@@ -45,6 +45,8 @@
         name = 'libxgboost.dylib'
     elif system() == 'Windows':
         name = 'xgboost.dll'
+    elif system() == 'AIX':
+        name = 'libxgboost.a'
     return name

# diff -u xgboost/libpath.py_orig xgboost/libpath.py
--- xgboost/libpath.py_orig     2023-05-30 02:07:56.190026737 +0000
+++ xgboost/libpath.py  2023-05-30 02:21:00.651401005 +0000
@@ -50,6 +50,8 @@
         dll_path = [os.path.join(p, 'libxgboost.dylib') for p in dll_path]
     elif sys.platform == 'cygwin':
         dll_path = [os.path.join(p, 'cygxgboost.dll') for p in dll_path]
+    elif sys.platform.startswith('aix'):
+        dll_path = [os.path.join(p, 'libxgboost.a') for p in dll_path]
 
     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]
 

# diff -u xgboost/core.py_orig xgboost/core.py
--- xgboost/core.py_orig        2023-05-30 02:07:56.180025453 +0000
+++ xgboost/core.py     2023-05-30 02:21:22.261566474 +0000
@@ -172,7 +172,7 @@
             os.environ["PATH"] = os.pathsep.join(
                 pathBackup + [os.path.dirname(lib_path)]
             )
-            lib = ctypes.cdll.LoadLibrary(lib_path)
+            lib = ctypes.cdll.LoadLibrary(lib_path + '(libxgboost.so)')
             lib_success = True
         except OSError as e:
             os_error_list.append(str(e))

Then inside python-package directory , run python3 -m pip install . 

I would recommend you to use a custom path to install it , python3 -m pip install . --root <custompath>
Then do export PYTHONPATH=<custompath>/opt/freeware/lib64/python3.7/site-packages

Hi Ranjit,

I have followed the steps you suggested. But got following errors when doing step 7: running 'python3.7 setup.py install'. Please help.

Thanks

Mi

-- Check for working CXX compiler: /opt/freeware/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/freeware/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- CMake version 3.22.0
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT - Failed
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT - Success
-- xgboost VERSION: 1.6.2
-- Setting build type to 'Release' as none was specified.
-- Looking for pthread.h
-- Looking for pthread.h - not found
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:151 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
WARNING:XGBoost build_ext:Disabling OpenMP support.
INFO:XGBoost build_ext:Run CMake command: ['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']
-- CMake version 3.22.0
-- xgboost VERSION: 1.6.2
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:151 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 197, in build_cmake_extension
    self.build(src_dir, build_dir, gen, build_tool, use_omp=1)
  File "setup.py", line 134, in build
    subprocess.check_call(cmake_cmd, cwd=build_dir)
  File "/opt/freeware/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=1', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 372, in <module>
    url='https://github.com/dmlc/xgboost')
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 307, in run
    super().run()
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/install.py", line 65, in run
    orig.install.run(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 205, in build_extension
    self.build_cmake_extension()
  File "setup.py", line 200, in build_cmake_extension
    self.build(src_dir, build_dir, gen, build_tool, use_omp=0)
  File "setup.py", line 134, in build
    subprocess.check_call(cmake_cmd, cwd=build_dir)
  File "/opt/freeware/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']' returned non-zero exit status 1.
root [bastdctdv64] /tmp/new-libs-1/xgboost-1.6.2:

Hi Wang,

So direct pip installation will not work. We need to make some source code change to make it compatible with AIX.  
Try below steps and see if it's working for you. 

First download the source from below link
https://files.pythonhosted.org/packages/25/e0/f00261d3357eafbe7c657de718abd3c0f6613633a38d05350f63ee099cde/xgboost-1.6.2.tar.gz

1. Extract it : tar -xvf xgboost-1.6.2.tar.gz
2. cd xgboost-1.6.2
3. Export the env which I had shared previously.
4. Run 'python3.7 setup.py install'. Same below error will be reported.

    self.build_cmake_extension()

  File "setup.py", line 153, in build_cmake_extension

    os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))

  File "setup.py", line 48, in lib_name

    return name

UnboundLocalError: local variable 'name' referenced before assignment.

To fix this , we need to modify the source as explained. Make sure to use proper spacing for modifying below source file as these are python code and spacing matters. Use 4 spacing for modifying the code. 

5. Take a backup of setup.py and modify setup.py with below changes.

# diff -u setup.py.orig setup.py

--- setup.py.orig       2023-05-29 01:40:36.295265022 +0000

+++ setup.py    2023-05-29 01:43:15.045066128 +0000

@@ -45,6 +45,8 @@

         name = 'libxgboost.dylib'

     elif system() == 'Windows':

         name = 'xgboost.dll'

+    elif system() == 'AIX':

+        name = 'libxgboost.a'

     return name

6. Take a backup of xgboost/libpath.py  and modify  xgboost/libpath.py with below change

(0) root @ pokndd3: /ranjit/xgboost-1.6.2/xgboost

# diff -u libpath.py.orig libpath.py

--- libpath.py.orig     2023-05-29 01:57:12.505062007 +0000

+++ libpath.py  2023-05-29 02:16:33.611085204 +0000

@@ -50,6 +50,8 @@

         dll_path = [os.path.join(p, 'libxgboost.dylib') for p in dll_path]

     elif sys.platform == 'cygwin':

         dll_path = [os.path.join(p, 'cygxgboost.dll') for p in dll_path]

+    elif sys.platform.startswith('aix'):

+        dll_path = [os.path.join(p, 'libxgboost.so') for p in dll_path]

     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]

7. python3.7 setup.py install   (This should pass now)
   
   
8. Execute below:

cp /opt/freeware/lib64/python3.7/site-packages/xgboost/lib/libxgboost.a /opt/freeware/lib

cd /opt/freeware/lib/

ar -x libxgboost.a

9. Now, xgboost can be imported.

# python3

Python 3.7.15 (default, Dec 15 2022, 03:08:16)

[GCC 8.3.0] on aix7

Type "help", "copyright", "credits" or "license" for more information.

>>> import xgboost

------------------------------
Ranjit Ranjan

Original Message:
Sent: Fri May 26, 2023 04:10 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit / Ayappan,

After I reinstalled the default setuptools (47.1.0), now I got the similar error when installing xgboost as you did. Please help to resolve the issue:

  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 203, in build_extension
        self.build_cmake_extension()
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 153, in build_cmake_extension
        os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 48, in lib_name
        return name
    UnboundLocalError: local variable 'name' referenced before assignment

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Fri May 26, 2023 07:40 AM
From: Ayappan P
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,
Looks like something is wrong with your setuptools. 
You can remove the existing setuptools and install it freshly. 
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools*dist-info

python3 -m ensurepip
python3 -m pip install --upgrade setuptools

Then you can try installing xgboost. 

------------------------------
Ayappan P

Original Message:
Sent: Wed May 24, 2023 05:22 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Wed May 24, 2023 10:48 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Tue May 23, 2023 02:58 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

------------------------------
Mi Wang

Original Message:
Sent: Mon May 22, 2023 03:27 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Thu May 18, 2023 02:50 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Thu May 18, 2023 12:57 PM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Wed May 17, 2023 12:16 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Tue May 16, 2023 02:28 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Mon May 15, 2023 02:37 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

------------------------------
Mi Wang
------------------------------

Hi Ayappan,

I am wondering if there is some workaround to avoid using those git commands? Our servers don't have internet connection due to the security control measures on them.

Thanks

Mi

Can you try the below steps

git clone https://github.com/dmlc/xgboost
cd xgboost
git checkout v1.6.2
git submodule init
git submodule update
mkdir build
cd build
export CC="gcc -maix64 -pthread"
export CXX="g++ -maix64 -pthread"
export OBJECT_MODE=64
cmake ..

cd ../python-package

Modify below three files

# diff -u setup.py_orig setup.py
--- setup.py_orig       2023-05-30 02:07:56.160027553 +0000
+++ setup.py    2023-05-30 02:19:35.710912296 +0000
@@ -45,6 +45,8 @@
         name = 'libxgboost.dylib'
     elif system() == 'Windows':
         name = 'xgboost.dll'
+    elif system() == 'AIX':
+        name = 'libxgboost.a'
     return name

# diff -u xgboost/libpath.py_orig xgboost/libpath.py
--- xgboost/libpath.py_orig     2023-05-30 02:07:56.190026737 +0000
+++ xgboost/libpath.py  2023-05-30 02:21:00.651401005 +0000
@@ -50,6 +50,8 @@
         dll_path = [os.path.join(p, 'libxgboost.dylib') for p in dll_path]
     elif sys.platform == 'cygwin':
         dll_path = [os.path.join(p, 'cygxgboost.dll') for p in dll_path]
+    elif sys.platform.startswith('aix'):
+        dll_path = [os.path.join(p, 'libxgboost.a') for p in dll_path]
 
     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]
 

# diff -u xgboost/core.py_orig xgboost/core.py
--- xgboost/core.py_orig        2023-05-30 02:07:56.180025453 +0000
+++ xgboost/core.py     2023-05-30 02:21:22.261566474 +0000
@@ -172,7 +172,7 @@
             os.environ["PATH"] = os.pathsep.join(
                 pathBackup + [os.path.dirname(lib_path)]
             )
-            lib = ctypes.cdll.LoadLibrary(lib_path)
+            lib = ctypes.cdll.LoadLibrary(lib_path + '(libxgboost.so)')
             lib_success = True
         except OSError as e:
             os_error_list.append(str(e))

Then inside python-package directory , run python3 -m pip install . 

I would recommend you to use a custom path to install it , python3 -m pip install . --root <custompath>
Then do export PYTHONPATH=<custompath>/opt/freeware/lib64/python3.7/site-packages

Hi Ranjit,

I have followed the steps you suggested. But got following errors when doing step 7: running 'python3.7 setup.py install'. Please help.

Thanks

Mi

-- Check for working CXX compiler: /opt/freeware/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/freeware/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- CMake version 3.22.0
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT - Failed
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT - Success
-- xgboost VERSION: 1.6.2
-- Setting build type to 'Release' as none was specified.
-- Looking for pthread.h
-- Looking for pthread.h - not found
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:151 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
WARNING:XGBoost build_ext:Disabling OpenMP support.
INFO:XGBoost build_ext:Run CMake command: ['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']
-- CMake version 3.22.0
-- xgboost VERSION: 1.6.2
CMake Error at /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /opt/freeware/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/freeware/share/cmake-3.22/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:151 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeOutput.log".
See also "/tmp/new-libs-1/xgboost-1.6.2/build/temp.aix-7.2-3.7/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 197, in build_cmake_extension
    self.build(src_dir, build_dir, gen, build_tool, use_omp=1)
  File "setup.py", line 134, in build
    subprocess.check_call(cmake_cmd, cwd=build_dir)
  File "/opt/freeware/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=1', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 372, in <module>
    url='https://github.com/dmlc/xgboost')
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 307, in run
    super().run()
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/install.py", line 65, in run
    orig.install.run(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 205, in build_extension
    self.build_cmake_extension()
  File "setup.py", line 200, in build_cmake_extension
    self.build(src_dir, build_dir, gen, build_tool, use_omp=0)
  File "setup.py", line 134, in build
    subprocess.check_call(cmake_cmd, cwd=build_dir)
  File "/opt/freeware/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', 'xgboost', '-GUnix Makefiles', '-DUSE_OPENMP=0', '-DUSE_CUDA=0', '-DUSE_NCCL=0', '-DBUILD_WITH_SHARED_NCCL=0', '-DHIDE_CXX_SYMBOLS=1', '-DUSE_HDFS=0', '-DUSE_AZURE=0', '-DUSE_S3=0', '-DPLUGIN_DENSE_PARSER=0']' returned non-zero exit status 1.
root [bastdctdv64] /tmp/new-libs-1/xgboost-1.6.2:

Hi Wang,

So direct pip installation will not work. We need to make some source code change to make it compatible with AIX.  
Try below steps and see if it's working for you. 

First download the source from below link
https://files.pythonhosted.org/packages/25/e0/f00261d3357eafbe7c657de718abd3c0f6613633a38d05350f63ee099cde/xgboost-1.6.2.tar.gz

1. Extract it : tar -xvf xgboost-1.6.2.tar.gz
2. cd xgboost-1.6.2
3. Export the env which I had shared previously.
4. Run 'python3.7 setup.py install'. Same below error will be reported.

    self.build_cmake_extension()

  File "setup.py", line 153, in build_cmake_extension

    os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))

  File "setup.py", line 48, in lib_name

    return name

UnboundLocalError: local variable 'name' referenced before assignment.

To fix this , we need to modify the source as explained. Make sure to use proper spacing for modifying below source file as these are python code and spacing matters. Use 4 spacing for modifying the code. 

5. Take a backup of setup.py and modify setup.py with below changes.

# diff -u setup.py.orig setup.py

--- setup.py.orig       2023-05-29 01:40:36.295265022 +0000

+++ setup.py    2023-05-29 01:43:15.045066128 +0000

@@ -45,6 +45,8 @@

         name = 'libxgboost.dylib'

     elif system() == 'Windows':

         name = 'xgboost.dll'

+    elif system() == 'AIX':

+        name = 'libxgboost.a'

     return name

6. Take a backup of xgboost/libpath.py  and modify  xgboost/libpath.py with below change

(0) root @ pokndd3: /ranjit/xgboost-1.6.2/xgboost

# diff -u libpath.py.orig libpath.py

--- libpath.py.orig     2023-05-29 01:57:12.505062007 +0000

+++ libpath.py  2023-05-29 02:16:33.611085204 +0000

@@ -50,6 +50,8 @@

         dll_path = [os.path.join(p, 'libxgboost.dylib') for p in dll_path]

     elif sys.platform == 'cygwin':

         dll_path = [os.path.join(p, 'cygxgboost.dll') for p in dll_path]

+    elif sys.platform.startswith('aix'):

+        dll_path = [os.path.join(p, 'libxgboost.so') for p in dll_path]

     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]

7. python3.7 setup.py install   (This should pass now)
   
   
8. Execute below:

cp /opt/freeware/lib64/python3.7/site-packages/xgboost/lib/libxgboost.a /opt/freeware/lib

cd /opt/freeware/lib/

ar -x libxgboost.a

9. Now, xgboost can be imported.

# python3

Python 3.7.15 (default, Dec 15 2022, 03:08:16)

[GCC 8.3.0] on aix7

Type "help", "copyright", "credits" or "license" for more information.

>>> import xgboost

------------------------------
Ranjit Ranjan

Original Message:
Sent: Fri May 26, 2023 04:10 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit / Ayappan,

After I reinstalled the default setuptools (47.1.0), now I got the similar error when installing xgboost as you did. Please help to resolve the issue:

  File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/opt/freeware/lib64/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 203, in build_extension
        self.build_cmake_extension()
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 153, in build_cmake_extension
        os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
      File "/tmp/pip-install-jto30iwh/xgboost/setup.py", line 48, in lib_name
        return name
    UnboundLocalError: local variable 'name' referenced before assignment

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Fri May 26, 2023 07:40 AM
From: Ayappan P
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,
Looks like something is wrong with your setuptools. 
You can remove the existing setuptools and install it freshly. 
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools
rm -r /opt/freeware/lib/python3.7/site-packages/setuptools*dist-info

python3 -m ensurepip
python3 -m pip install --upgrade setuptools

Then you can try installing xgboost. 

------------------------------
Ayappan P

Original Message:
Sent: Wed May 24, 2023 05:22 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I see following in the build_py.py. Maybe the setuptools's version is too old in my system (it's 59.2.0). What's the setuptools version in your system? Is your setuptools package also from PYPI?

try:
    from setuptools.lib2to3_ex import Mixin2to3
except ImportError:

    class Mixin2to3:
        def run_2to3(self, files, doctests=True):
            "do nothing"

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Wed May 24, 2023 10:48 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Wang,

Suspecting  some python configuration issue in your system .  
I got below error while running 'pip3 install xgboost==1.6.2'


    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/opt/freeware/lib64/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 203, in build_extension
      self.build_cmake_extension()
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 153, in build_cmake_extension
      os.path.join(CURRENT_DIR, os.path.pardir, 'lib', lib_name()))
    File "/tmp/pip-install-qjq6vaku/xgboost_b8e68e1351b843aab1cda2c4f932547c/setup.py", line 48, in lib_name
      return name
  UnboundLocalError: local variable 'name' referenced before assignment
  [end of output]

Could you try from other system and confirm ? 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Tue May 23, 2023 02:58 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I have tried to install xgboost 1.6.2 on python 3.7 and got the following error. Is there a way to get around this 'convert_2to3_doctests' attribute?

Thanks

Mi

copying xgboost/CMakeLists.txt -> build/lib.aix-7.1-3.7/xgboost
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-76li2fpj/xgboost/setup.py", line 370, in <module>
      url='https://github.com/dmlc/xgboost')
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/freeware/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/freeware/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/freeware/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 58, in run
      self.build_package_data()
    File "/opt/freeware/lib/python3.7/site-packages/setuptools/command/build_py.py", line 132, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for xgboost
  Running setup.py clean for xgboost
Failed to build xgboost

------------------------------
Mi Wang

Original Message:
Sent: Mon May 22, 2023 03:27 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi

As per https://pypi.org/project/xgboost/1.6.2/ , this version requirement is python 3.7. Try this version once and let us know. 

------------------------------
Ranjit Ranjan

Original Message:
Sent: Thu May 18, 2023 02:50 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

1. I added the 3 export commands in the file xgboost/Makefile. I didn't run them in interactive session. Maybe I should run them in the shell session instead?
2. I tried xgboost on an old version 0.82 on python 3.7 (AIX Toolbox one) by using python's pip tool. I am not which version Python you have tried this xgboost 1.7.5? Will it work on python 3.7 (I tried this before and got some strange errors)?

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Thu May 18, 2023 12:57 PM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

• How did you run above export commands . 

          Did you run above export commands in shell session from where you are trying to compile   Or some other way ?

          You can run these from the shell session and verify using the echo command like below. It should work. 

export OBJECT_MODE=64
export CC="gcc  -maix64 -pthread"
export CXX="g++ -maix64 -pthread"

# echo $CC
gcc -maix64 -pthread

•  I thought to try and compile c++ shared library , so I used xgboost version 1.7.5.  I see below compilation failure. To fix this I made some code changes also . 

/home/ranjit/xgboost/include/xgboost/collective/socket.h:294:41: error: 'SO_DOMAIN' was not declared in this scope; did you mean 'DOMAIN'?

  294 |         getsockopt(handle_, SOL_SOCKET, SO_DOMAIN, reinterpret_cast<char *>(&domain), &len), 0);

      |                                         ^~~~~~~~~

/home/ranjit/xgboost/include/xgboost/base.h:73:54: note: in definition of macro 'XGBOOST_EXPECT'

   73 | #define XGBOOST_EXPECT(cond, ret)  __builtin_expect((cond), (ret))

      |                                                      ^~~~

/home/ranjit/xgboost/include/xgboost/collective/socket.h:293:5: note: in expansion of macro 'xgboost_CHECK_SYS_CALL'

  293 |     xgboost_CHECK_SYS_CALL(

      |     ^~~~~~~~~~~~~~~~~~~~~~

make[2]: *** [src/CMakeFiles/objxgboost.dir/build.make:118: src/CMakeFiles/objxgboost.dir/collective/socket.cc.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:264: src/CMakeFiles/objxgboost.dir/all] Error 2

make: *** [Makefile:156: all] Error 2

• To fix above compile error ,  I have modified include/xgboost/collective/socket.h. Below is git diff of this file.

# git diff include/xgboost/collective/socket.h

diff --git a/include/xgboost/collective/socket.h b/include/xgboost/collective/socket.h

index 33d14fe8..4f9dc565 100644

--- a/include/xgboost/collective/socket.h

+++ b/include/xgboost/collective/socket.h

@@ -287,7 +287,7 @@ class TCPSocket {

 #elif defined(__APPLE__)

     return domain_;

 #elif defined(__unix__)

-#ifndef __PASE__

+#if defined(__PASE__) && !defined(AIX)

     std::int32_t domain;

     socklen_t len = sizeof(domain);

     xgboost_CHECK_SYS_CALL

Hope, you understand  above change and make out that which line is removed and which is added.  

After this changes I am able to build shared object and executable .

[ 93%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/allreduce_base.cc.o

[ 95%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/rabit_c_api.cc.o

[ 96%] Building CXX object src/CMakeFiles/objxgboost.dir/__/rabit/src/engine.cc.o

[ 96%] Built target objxgboost

[ 97%] Building CXX object CMakeFiles/runxgboost.dir/src/cli_main.cc.o

[ 98%] Linking CXX executable xgboost

[ 98%] Built target runxgboost

[100%] Linking CXX shared library lib/libxgboost

[100%] Built target xgboost

Let me know if you need any info regarding above.

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Wed May 17, 2023 12:16 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi Ranjit,

I add the settings and redo the PIP installation. Now I got the following errors. It seem the CC variable can only be pointing to some Unix executable. Can you help?

   Makefile:31: MAKE [make] - checked OK
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api_error.o src/c_api/c_api_error.cc >build/c_api/c_api_error.d
    'g++ -maix64 -pthread' -DDMLC_LOG_CUSTOMIZE=1 -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude   -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -fPIC -fopenmp -MM -MT build/c_api/c_api.o src/c_api/c_api.cc >build/c_api/c_api.d
    /bin/sh: g++ -maix64 -pthread:  not found
    /bin/sh: g++ -maix64 -pthread:  not found
    make: *** [Makefile:147: build/learner.o] Error 127
..........

Thanks

Mi

------------------------------
Mi Wang

Original Message:
Sent: Tue May 16, 2023 02:28 AM
From: Ranjit Ranjan
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

HI,

What are the steps used for compiling ? Please share that. 
Try setting below in env and then try.

export OBJECT_MODE=64

export CC='gcc  -maix64 -pthread'

export CXX='g++ -maix64 -pthread'

Thanks
Ranjit

------------------------------
Ranjit Ranjan

Original Message:
Sent: Mon May 15, 2023 02:37 PM
From: Mi Wang
Subject: Issue with C / C++ Compiling during the PIP installation for Xgboost

Hi,

I got the following errors when trying to do pip install for Xgboost on python 3.7. Can AIX Toolbox experts help?

1. "dmlc-core/include/dmlc/thread_local.h:76:8: error: 'mutex' in namespace 'std' does not name a type"
Is this related to C++ setup? We already have libstdc++ and libstdc++10 installed.

2. "rabit/include/rabit/./internal/../../dmlc/./logging.h:288:19: error: 'backtrace' was not declared in this scope"
I see the following in base.sh to define the controlling variable to enable backtrace. Is it safe to comment out that portion? (considering backtrace is only for linux):
"#if (!defined(DMLC_LOG_STACK_TRACE) \
     && defined(__GNUC__) && !defined(__MINGW32__) \
     && !defined(__sun) && !defined(__SVR4))
#define DMLC_LOG_STACK_TRACE 1
#endif
"
3. "make: cc: No such file or directory":
I see in the general makefile the following environment settings (only for Darwin and Linux). Can you provide one for AIX so that can use it to replace following?
ifeq ($(OS), Darwin)
ifndef CC
export CC = $(if $(shell which clang), clang, gcc)
endif
ifndef CXX
export CXX = $(if $(shell which clang++), clang++, g++)
endif
else
# linux defaults
ifndef CC
export CC = gcc
endif
ifndef CXX
export CXX = g++
endif
endif

Thanks

Mi

------------------------------
Mi Wang
------------------------------