Open Source Development

Power Open Source Development

Explore the open source tools and capabilities for building and deploying modern applications on IBM Power platforms including AIX, IBM i, and Linux.


#Power


#Power

 View Only

  • 1.  AIX 7.2 binary core dumps on 7.3

    Posted 4 days ago

    Hi all,

    bash-5.2$ oslevel -s
    7200-04-01-1939

    We build a pyinstaller binary, the python script makes use of logging and pyzmq 27.1.0

    The issue is the binary built on 7.2 core dumps on 7.3 bash-5.2# Invalid argument (src/io_thread.cpp:68)

    I was suggested by the folks at pyinstaller to use -D_THREAD_SAFE, -D_THREAD_SAFE_ERRNO but the result is same.

    Below is the pyinstaller ticket link with sample code used to produce the error/issue.

    Specify lib-dynload on AIX 7.2 pyinstaller==6.15.0 · Issue #9241 · pyinstaller/pyinstaller

    GitHub remove preview
    Specify lib-dynload on AIX 7.2 pyinstaller==6.15.0 · Issue #9241 · pyinstaller/pyinstaller
    Hi We have python 3.11.3 built and has its own /opt/pyenv/versions/3.11.3/lib/python3.11/lib-dynload/ I then create a venv and install pyinstaller The python script is just import ssl print("Hello World") I create a binary using pyinstal...
    View this on GitHub >

     

    The same code compiles fine when building on AIX 7.3



    ------------------------------
    Avinesh Benjamin
    ------------------------------


  • 2.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 4 days ago

    @Ayappan P, @Aditya Kamath or other IBM employee please let me know if we can create ticket with this for IBM or it can be handled only by this forum since it involves open source components?



    ------------------------------
    Avinesh Benjamin
    ------------------------------

    Original Message


  • 3.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 4 days ago

    Hi,

    We need more details, for example, what CFLAGS and CXXFLAGS you had used to build, and so on. In AIX, Python modules are recommended to be built in 64-bit mode, though you can also build in 32 bit mode.

    Can you try like this?

    pip3.12 cache purge

    export CXX="g++ -pthread"
    export CXXFLAGS=-maix64
    export OBJECT_MODE=64
    export LDFLAGS="-maix64"
    export CC="gcc -pthread"
    export CFLAGS=-maix64
    pip3.12 install pyinstaller 

    And then try??

    Note: if you use Python 3.11, then change all commands to pip3.11. We also want to know which Python 3.11 are you using?
    Also a common mistake that most make is using the AIX-based Python instead of the toolbox Python. 

    So there could be a variety of reasons for not working. We need more details.



    ------------------------------
    Aditya Kamath
    ------------------------------

    Original Message


  • 4.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 3 days ago

    Hi,

    OBJECT_MODE=64

    AR=ar -X64

    (venv) bash-5.2$ pip list
    Package                   Version
    ------------------------- -------
    altgraph                  0.17.4
    packaging                 25.0
    pip                       22.3.1
    pyinstaller               6.15.0
    pyinstaller-hooks-contrib 2025.9
    pyzmq                     27.1.0
    setuptools                65.5.0
    (venv) bash-5.2$ python --version
    Python 3.11.3

    Python 3.11.3 is built by us, we use pyenv to maintain multiple python versions.

    For pyinstaller I did the below:

    bash-5.2$ /opt/IBM/xlC/16.1.0/bin/xlclang --version
    IBM XL C/C++ for AIX, V16.1.0  (5725-C72, 5765-J12)
    Version: 16.01.0000.0021

    CC=/opt/IBM/xlC/16.1.0/bin/xlclang

    CXX=/opt/IBM/xlC/16.1.0/bin/xlclang++

    pip download -d venv/ pyinstaller==6.15.0

    then did export CFLAGS="-D_THREAD_SAFE, -D_THREAD_SAFE_ERRNO" as suggested by pyinstaller folks

    cd pyinst_debug/7_3/venv/pyinstaller-6.15.0

    bootloader/waf distclean all -vv

    For pyzmq the below is changed

     export CXX="/opt/freeware/bin/g++"
     export CFLAGS="-maix64 -pthread -fPIC"
     export CXXFLAGS="-maix64 -pthread -fPIC" 

    export AR="ar -X64"

    export ORIG_PATH=$PATH

    export PATH=$PATH:/opt/freeware/bin  
    export LDFLAGS="-Wl,-bnoipath /opt/pyenv/versions/3.11.3/lib/libpython3.11.so"  
    $PIP_CMD install pyzmq==27.1.0 --no-binary pyzmq --no-cache-dir
    export LDFLAGS=""
    export PATH=$ORIG_PATH

    gcc is version 10

    We use the same steps on AIX 7.3 except for setting CFLAGS before pyinstaller

    Even the python pyenv is copied and pasted on the AIX 7.3 setup from the 7.2 and the binary works without issues.

    libzmq is downloaded from opensource toolbox

    To reproduce the issue you can copy the code I shared in the pyinstaller ticket and run the pub code which connects to pyinstaller binary, it needs to be run multiple times for the core dump to occur

    sometimes close to 30 hours.



    ------------------------------
    Avinesh Benjamin
    ------------------------------

    Original Message


  • 5.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 3 days ago

    Hi,

    First of all, it is not a good idea to mix compilers for different modules since we have seen in the past that at times, due to compatibility issues, Python modules can core dump.

    So I would advice sticking to GCC or only openXL or XLC. 

    The next thing I want to ask is, since you mentioned you build Python on your own, what are the flags/compilers you used to do the same? Did you build a 64-bit or a 32 bit? What is output of file /opt/pyenv/versions/3.11.3/lib/libpython3.11.so?

    Lastly,

    >pip download -d venv/ pyinstaller==6.15.0

    >then did export CFLAGS="-D_THREAD_SAFE, -D_THREAD_SAFE_ERRNO" as suggested by pyinstaller folks

    >cd pyinst_debug/7_3/venv/pyinstaller-6.15.0

    Did you pass -q64 here?

    I would recommend you stick to one compiler for now, which is GCC, and build these one by one. 
    pip3.11 cache purge
    pip3.11 uninstall pyzmq pyinstaller
    export CXX="g++ -pthread"
    export CXXFLAGS=-maix64
    export OBJECT_MODE=64
    export LDFLAGS="-maix64"
    export CC="gcc -pthread"
    export CFLAGS=-maix64
    pip3.11 install pyinstaller
    pip3.11 install pyzmq 


    Your flags for pyzmq is correct. So that should have been correctly built. Let us go one by one, and let me know the answer to all the questions, and check if everything built with GCC works or not?. 



    ------------------------------
    Aditya Kamath
    ------------------------------

    Original Message


  • 6.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 2 days ago

    Hi,

    Thanks for the response

    From pyinstaller log

    /opt/IBM/xlC/16.1.0/bin/xlclang', '-q64', '../test.c', '-c', '-o/disks/drm_lonestar/benjaa2/pyinst_debug/7_3/venv/pyinstaller-6.15.0/bootloader/bu
    ild/.conf_check_5d7dca335ebac7b3d0e820404365b5c9/testbuild/test.c.1.o

    On when using xlclang setting OBJECT_MODE=64 ensures 64 bit builds and also setting the linker field as needed.

    From public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/SPECS/gettext-0.21-2.spec

    public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/SPECS/python3.11-3.11.9-1.spec

    export LDFLAGS="-Wl,-blibpath:/opt/freeware/lib64:/opt/freeware/lib:/usr/lib"

    We were linking both -L/usr/lib and -L/opt/freeware in above pages are we not mixing xlclang and gcc in that case as well?

    We upgraded to 7200-05-10-2520 but now facing issues building python keep getting ERROR: Undefined symbol: .libintl_dcgettext

    Following steps from here public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/SPECS/python3.11-3.11.9-1.spec

    We have both yum and dnf is our machine broken post upgrade?



    ------------------------------
    Avinesh Benjamin
    ------------------------------

    Original Message


  • 7.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 2 days ago

    @Avinesh Benjamin

    So why are you building Python 3.11 on your own? 
    You also mentioned DNF is not working. Why is it so? What happens when you do dnf update?
    I also want to know why you are using xlc compiler? I will install and check with xlc as well on what is going on.

    Coming to the issue I am currently in oslevel 7.2 TL 5. 

    # cat test.py 
    import ssl 
    print("Hello World")
    ((my_env) ) 

    I used the same script as yours in the GitHub issue. 

    Using the GCC compiler using below options/steps I was able to install pyinstaller.

    export CXX="g++ -pthread"
    export CXXFLAGS=-maix64
    export OBJECT_MODE=64
    export LDFLAGS="-maix64"
    export CC="gcc -pthread"
    export CFLAGS=-maix64
    pip3.11 install pyinstaller
    Then ran pyinstall test.py and it installed as shown below

    50999 INFO: Building COLLECT COLLECT-00.toc completed successfully.
    51007 INFO: Build complete! The results are available in: /dist

    After copying the dist directory in 7.3 LPAR and running it there, I was able to run. See below:
    # /home/dist/test/test 
    Hello World

    While I try with xlc, I think there is some issue with your environment which we need to get to one by one. Also share me the detailed steps if it is any different from what you mentioned so far for xlc based pyinstaller installed. Ideally if the environment is correct we should get it working. 



    ------------------------------
    Aditya Kamath
    ------------------------------

    Original Message


  • 8.  RE: AIX 7.2 binary core dumps on 7.3

    Posted yesterday

    Hi @Aditya Kamath

    As already mentioned in the pyinstaller link and this thread the problem occurs in the libzmq where an assertion fails as errno is einval leading to the binary core dumping, and as already mentioned the issue takes 10 hours sometimes 1 day for the core dump to occur, the issue was never with a simple hello world or print statement binary.

    import time
import logging
from zmq.auth.thread import ThreadAuthenticator
import zmq.auth
import os


def init_log() -> logging.Logger:
    log_filename = f"sub_{int(time.time())}.log"
    logger:logging.Logger = logging.getLogger(__name__)
    logger.setLevel(logging.INFO)
    file_handler = logging.FileHandler(log_filename)
    file_handler.setLevel(logging.DEBUG)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    return logger


def create_certs():
    """Create zmq certificates."""
    if not os.access('./certs', os.R_OK):
        os.mkdir('./certs')
        zmq.auth.create_certificates('./certs','pyinst')

with zmq.Context.instance() as zmq_context:
    logger = init_log()
    create_certs()
    logger.info('Certs created!')
    auth = ThreadAuthenticator(zmq_context)
    auth.start()
    auth.deny('')
    auth.configure_curve(domain='*', location=f'./certs')
    zmq_socket:zmq.Socket = zmq_context.socket(zmq.SUB)
    msgsvc_public_key, msgsvc_secret_key = zmq.auth.load_certificate('./certs/pyinst.key_secret')
    zmq_socket.curve_publickey = msgsvc_public_key
    zmq_socket.curve_secretkey = msgsvc_secret_key
    zmq_socket.curve_server = True
    # zmq_socket.setsockopt(zmq.LINGER, 0)
    zmq_socket.setsockopt_string(zmq.SUBSCRIBE, '')
    zmq_socket.bind("tcp://127.0.0.1:5000")
    # print(msgsvc_public_key, msgsvc_secret_key)
    poller = zmq.Poller()
    poller.register(zmq_socket, zmq.POLLIN)

    while True:
        e = dict(poller.poll(200))
        sock = dict(e)
        if sock and sock.get(zmq_socket) == zmq.POLLIN:
            message = zmq_socket.recv_multipart()
            if message:
                logger.info(message)
    auth.stop()

    pyinstaller --clean -y --onefile --runtime-tmpdir ./ -n sub sub.py

    and here is the C program

    #include <iostream>
#include <zmq.h>
#include <string>
#include <cstring>
#include <unistd.h>
#include <memory>

const char* PUBLIC_KEY = "4>4Jf8L2v6+F>eER?>4QLym>wFU5JTJ#clJJK%QJ";
const char* SERVER_KEY = "4>4Jf8L2v6+F>eER?>4QLym>wFU5JTJ#clJJK%QJ";
const char* PRIVATE_KEY = "UC<^r@/wI>fpJY$/8WomOF-EJX8Eq{xhM8{Yl#4.";


void print_error(const char *const ptr)
{
    std::cout << ptr << "\n";
}

int main()
{
    void *m_zmq_ctx;
    void *m_socket;
    m_zmq_ctx = zmq_ctx_new();
    const int backlog = 10;

    if (m_zmq_ctx == NULL)
    {
        print_error("Unable to create context");
        exit(1);
    }
    if (zmq_ctx_set(m_zmq_ctx, ZMQ_BLOCKY, false) != 0)
    {
        print_error("Unable to set context to non-blocking mode");
        exit(1);
    }
    m_socket = zmq_socket(m_zmq_ctx, ZMQ_PUB);
    if (m_socket == NULL)
    {
        print_error("Unable to create socket");
        exit(1);
    }
    int ipv6=1;
    if (zmq_setsockopt(m_socket, ZMQ_IPV6, &ipv6, sizeof(ipv6))) {
        print_error("Unable to set socket option to ipv6");
        exit(1);
    }
    if (zmq_setsockopt(m_socket, ZMQ_CURVE_PUBLICKEY, PUBLIC_KEY, strlen(PUBLIC_KEY)))
    {
        print_error("Unable to set socket curve public key");
        exit(1);
    }
    if (zmq_setsockopt(m_socket, ZMQ_CURVE_SECRETKEY, PRIVATE_KEY, strlen(PRIVATE_KEY)))
    {
        print_error("Unable to set socket curve public key");
        exit(1);
    }

    if (zmq_setsockopt(m_socket, ZMQ_CURVE_SERVERKEY, SERVER_KEY, strlen(SERVER_KEY)))
    {
        print_error("Unable to set socket curve public key");
        exit(1);
    }

    if (zmq_connect(m_socket, "tcp://127.0.0.1:5000") != 0)
    {
        print_error("Unable to bind socket to portnum");
        exit(1);
    }
    for (int i = 0; i < 10; ++i)
    {
        std::string buffer = "Hello World " + std::to_string(i);
        std::cout << "Sending: " << buffer << "\n";
        if (zmq_send(m_socket, buffer.c_str(), buffer.size(), 0) == -1)
        {
            print_error("Unable to send message");
        }
        sleep(4);
    }
    std::cout << "Result of Socket close and context term: " << (zmq_close(m_socket) == 0 && zmq_ctx_term(m_zmq_ctx) == 0) << "\n";
}

    the above code is compiled with xlclang++ and libzmq built with xlclang++

    The sercret key and other vals in the above binary can be copied from certs/pyinst.key_secret

    You can run the sub binary as ./sub & (so it run in the background as long process) on the command line and then trigger pub (c++ code) via cron jobs run it every 5 minutes you should encounter the assertion error and core dump later.

    Please note both binaries need to be built on AIX 7.2 and run on AIX 7.3

    Regarding the above I also tried building with gcc-10 earlier and I recall facing core dump, I have trided using --onedir option to replace libintl and libzmq libraries copied from AIX 7.3 as well, If the binary is built using pyinstaller on AIX 7.2 it fails.

    Now for this part

    We upgraded to 7200-05-10-2520 but now facing issues building python keep getting ERROR: Undefined symbol: .libintl_dcgettext

    Following steps from here public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/SPECS/python3.11-3.11.9-1.spec

    We have both yum and dnf is our machine broken post upgrade? 

    Let me clarify this update was done 3 days back, and since the update we seem to have yum and dnf on the machine we are able to use dnf to install packages, but I don't follow why we need 2 package managers If something has gone regarding the upgrade/cleanup  the 2 don't need to be present on the machine you can let me know and I will ask our admin to rectify the issue.

    If needed, you can dnf install python3.11 on your AIX 7.2 machine and try the above steps you will still face a core dump.



    ------------------------------
    Avinesh Benjamin
    ------------------------------

    Original Message


  • 9.  RE: AIX 7.2 binary core dumps on 7.3

    Posted yesterday
    Edited by Avinesh Benjamin yesterday

    As to why we build python on our own we support multiple python versions and generally moved to the latest security patch from 3.11 or 3.12 if needed, along with other security updates. installing python versions via dnf or yum would defeat the purpose of using pyenv and managing multiple python installations.



    ------------------------------
    Avinesh Benjamin
    ------------------------------

    Original Message


  • 10.  RE: AIX 7.2 binary core dumps on 7.3

    Posted 3 days ago

    @Aditya Kamath Let me know if you need detailed build steps for the pyinstaller binary.



    ------------------------------
    Avinesh Benjamin
    ------------------------------

    Original Message


Related Content