AIX Open Source

 View Only
Expand all | Collapse all

Issue When Installing lxml for Python3 using pip

  • 1.  Issue When Installing lxml for Python3 using pip

    Posted Fri June 18, 2021 10:10 AM
    Hello all;
    I originally posted this in the AIX community, but, thankfully was directed to the more appropriate community here. Our PeopleSoft folks need lxml for Python3. We have been unable to locate Python3 with lxml for AIX so we are trying to use pip to install it using a download from the Python Package Index. Our AIX servers do not have internet access so pip cannot download it directly. AIX is at 7100-05-08-2114. We believe that we have installed the necessary prereqs, but the pip installation fails with a message to make sure they are installed. Also, where are the logs that we are to check for more info?

    # rpm -qa | grep python3
    libxml2-python3-2.9.10-1.ppc
    python3-tools-3.7.10-2.ppc
    python3-3.7.10-2.ppc
    python3-devel-3.7.10-2.ppc

    # rpm -qa | grep libxml2
    libxml2-python3-2.9.10-1.ppc
    libxml2-2.9.10-1.ppc
    libxml2-devel-2.9.10-1.ppc

    # rpm -qa | grep libxslt
    libxslt-1.1.34-1.ppc
    libxslt-devel-1.1.34-1.ppc

    # python -m pip install -v lxml-4.6.3.tar.gz --no-cache-dir
    Non-user install because site-packages writeable
    Created temporary directory: /tmp/pip-ephem-wheel-cache-1wk563bc
    Created temporary directory: /tmp/pip-req-tracker-ocf41tog
    Initialized build tracking at /tmp/pip-req-tracker-ocf41tog
    Created build tracker: /tmp/pip-req-tracker-ocf41tog
    Entered build tracker: /tmp/pip-req-tracker-ocf41tog
    Created temporary directory: /tmp/pip-install-206dmolc
    Processing ./lxml-4.6.3.tar.gz
    Created temporary directory: /tmp/pip-req-build-mynenchm
    Added file:///home/knighted/python3/lxml/lxml-4.6.3.tar.gz to build tracker '/tmp/pip-req-tracker-ocf41tog'
    Running setup.py (path:/tmp/pip-req-build-mynenchm/setup.py) egg_info for package from file:///<file path>/lxml/lxml-4.6.3.tar.gz
    Created temporary directory: /tmp/pip-pip-egg-info-kpbhtg7w
    Running command python setup.py egg_info
    Building lxml version 4.6.3.
    Building without Cython.
    Error: Please make sure the libxml2 and libxslt development packages are installed.
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    Exception information:
    Traceback (most recent call last):
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 488, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 550, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 530, in _generate_metadata
    details=self.name or "from {}".format(self.link)
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 73, in generate_metadata
    command_desc='python setup.py egg_info',
    File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 241, in call_subprocess
    raise InstallationError(exc_msg)
    pip._internal.exceptions.InstallationError: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    1 location(s) to search for versions of pip:
    * https://pypi.org/simple/pip/
    Fetching project page and analyzing links: https://pypi.org/simple/pip/
    Getting page https://pypi.org/simple/pip/
    Found index url https://pypi.org/simple
    Starting new HTTPS connection (1): pypi.org:443
    Could not fetch URL https://pypi.org/simple/pip/: connection error: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0xa0000000105ad88>: Failed to establish a new connection: [Errno 79] A remote host refused an attempted connect operation.')) - skipping
    Given no hashes to check 0 links for project 'pip': discarding no candidates
    Removed file:///home/knighted/python3/lxml/lxml-4.6.3.tar.gz from build tracker '/tmp/pip-req-tracker-ocf41tog'
    Removed build tracker: '/tmp/pip-req-tracker-ocf41tog'



    ------------------------------
    Ed Stuart
    ------------------------------


  • 2.  RE: Issue When Installing lxml for Python3 using pip

    Posted Fri June 18, 2021 12:33 PM
    Could it be that it cannot find the header files at /opt/freeware/include even though that is in the PATH? If so, how do we explicitly tell pip to look in that directory?

    ------------------------------
    Ed Stuart
    ------------------------------



  • 3.  RE: Issue When Installing lxml for Python3 using pip

    Posted Mon June 21, 2021 02:30 AM
    The libxslt installed is not from Toolbox. 
    rpm -qi libxslt ?
    Can you install libxslt & libxslt-devel from Toolbox and then try pip ?

    ------------------------------
    Ayappan P
    ------------------------------



  • 4.  RE: Issue When Installing lxml for Python3 using pip

    Posted Mon June 21, 2021 10:34 AM
    Thank-you Ayappan P; we installed the ones from the AIX Toolbox and are still getting the same error.

    # rpm -qi libxslt-devel-1.1.32-1.ppc
    Name : libxslt-devel
    Version : 1.1.32
    Release : 1
    Architecture: ppc
    Install Date: Mon Jun 21 09:12:31 CDT 2021
    Group : Development/Libraries
    Size : 2724077
    License : MIT
    Signature : (none)
    Source RPM : libxslt-1.1.32-1.src.rpm
    Build Date : Tue Mar 20 02:18:36 CDT 2018
    Build Host : pokndd5.pok.stglabs.ibm.com
    URL : http://xmlsoft.org/XSLT/
    Summary : Libraries, includes, etc. to embed the GNOME XSLT engine
    Description :
    This C library allows to transform XML files into other XML files
    (or HTML, text, ...) using the standard XSLT stylesheet transformation
    mechanism. To use it you need to have a version of libxml2 >= 2.9.5
    installed.

    If you are compiling a 32-bit program, no special compiler options are
    needed.

    If you are compiling a 64-bit program, you have to compile and link your
    application with "xlc_r -q64" or "gcc -maix64".

    # rpm -qi libxml2-devel-2.9.10-1.ppc
    Name : libxml2-devel
    Version : 2.9.10
    Release : 1
    Architecture: ppc
    Install Date: Mon Jun 21 09:04:16 CDT 2021
    Group : Development/Libraries
    Size : 9603061
    License : MIT
    Signature : (none)
    Source RPM : libxml2-2.9.10-1.src.rpm
    Build Date : Thu Nov 19 07:38:04 CST 2020
    Build Host : pokndd10.pok.stglabs.ibm.com
    Packager : IBM AIX Toolbox <https://ibm.biz/AIXToolbox>
    URL : http://xmlsoft.org/
    Bug URL : https://ibm.biz/aixoss_forum
    Summary : Libraries, includes, etc. to develop XML and HTML applications
    Description :
    Libraries, include files, etc you can use to develop XML applications.
    This library allows to manipulate XML files. It includes support
    to read, modify and write XML and HTML files. There is DTDs support
    this includes parsing and validation even with complex DtDs, either
    at parse time or later once the document has been modified. The output
    can be a simple SAX stream or and in-memory DOM like representations.
    In this case one can use the built-in XPath and XPointer implementation
    to select subnodes or ranges. A flexible Input/Output mechanism is
    available, with existing HTTP and FTP modules and combined to an
    URI library.

    If you are compiling a 32-bit program, no special compiler options are
    needed.

    If you are compiling a 64-bit program, you have to compile and link your
    application with "xlc -q64" or "gcc -maix64".

    # rpm -qi python3-devel-3.7.10-2.ppc
    Name : python3-devel
    Version : 3.7.10
    Release : 2
    Architecture: ppc
    Install Date: Mon Jun 21 09:16:26 CDT 2021
    Group : Development/Libraries
    Size : 783172
    License : Modified CNRI Open Source License
    Signature : (none)
    Source RPM : python3-3.7.10-2.src.rpm
    Build Date : Tue Jun 1 05:01:22 CDT 2021
    Build Host : pokndd10.pok.stglabs.ibm.com
    Relocations : /opt/freeware
    Packager : IBM AIX Toolbox <https://ibm.biz/AIXToolbox>
    URL : http://www.python.org/
    Bug URL : https://ibm.biz/aixoss_forum
    Summary : The libraries and header files needed for Python development.
    Description :
    The Python programming language s interpreter can be extended with
    dynamically loaded extensions and can be embedded in other programs.
    This package contains the header files and libraries needed to do
    these types of tasks.

    Install python-devel if you want to develop Python extensions. The
    python package will also need to be installed. You ll probably also
    want to install the python-docs package, which contains Python
    documentation.

    This version has been compiled with GCC.

    # sudo python -m pip install /home/knighted/python3/lxml/lxml-4.6.3 --no-cache-dir
    Password:
    WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
    Processing /home/knighted/python3/lxml/lxml-4.6.3
    ERROR: Command errored out with exit status 1:
    command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-b1ir5xps/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-b1ir5xps/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-e4ynlnhx
    cwd: /tmp/pip-req-build-b1ir5xps/
    Complete output (3 lines):
    Building lxml version 4.6.3.
    Building without Cython.
    Error: Please make sure the libxml2 and libxslt development packages are installed.
    ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
    Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping










    ------------------------------
    Ed Stuart
    ------------------------------



  • 5.  RE: Issue When Installing lxml for Python3 using pip
    Best Answer

    Posted Mon June 21, 2021 02:40 PM
    We were able to get around the pip issue by using the setup.py file with switches that point to the xslt and xml2 config files.
    sudo python setup.py install --with-xslt-config=/opt/freeware/bin/xslt-config  --with-xml2-config=/opt/freeware/bin/xml2-config

    It would be great if an lxml package for python3 would get added to the AIX Toolbox.

    ------------------------------
    Ed Stuart
    ------------------------------



  • 6.  RE: Issue When Installing lxml for Python3 using pip

    Posted Tue June 22, 2021 04:14 AM
    I think you haven't set your PATH to use "/opt/freeware/bin". 
    Otherwise this would have been smooth.

    ------------------------------
    Ayappan P
    ------------------------------



  • 7.  RE: Issue When Installing lxml for Python3 using pip

    Posted Wed October 04, 2023 03:54 AM

    We hit the same error during lxml installation and this package is required for the Ansible HMC Collection. Any idea how to install it smoothly?

    Processing ./lxml-4.9.3.tar.gz
      Preparing metadata (setup.py) ... error
      error: subprocess-exited-with-error
    
      python setup.py egg_info did not run successfully.
      exit code: 1
    
      [3 lines of output]
      Building lxml version 4.9.3.
      Building without Cython.
      Error: Please make sure the libxml2 and libxslt development packages are installed.
      [end of output]
    

    Installed packages:

    rpm -qa | grep -E "(python|libxml|libxslt)" | sort
    libxml2-2.10.3-1.ppc
    libxml2-python3-2.10.3-1.ppc
    libxslt-1.1.37-1.ppc
    python3-3.7.15-2.ppc
    python3-dateutil-2.8.0-1.noarch
    python3-devel-3.7.15-2.ppc
    python3-dnf-4.2.17-32_3.noarch
    python3-dnf-plugin-migrate-4.0.16-32_1.noarch
    python3-dnf-plugins-core-4.0.16-32_1.noarch
    python3-docs-3.7.15-2.ppc
    python3-gpg-1.13.1-32_2.ppc
    python3-hawkey-0.39.1-32_2.ppc
    python3-jinja2-2.10.3-1.noarch
    python3-jmespath-0.9.4-1.noarch
    python3-libcomps-0.1.11-32_1.ppc
    python3-libdnf-0.39.1-32_2.ppc
    python3-librepo-1.11.0-32_1.ppc
    python3-markupsafe-1.1.1-1.ppc
    python3-pyyaml-5.4.1.1-1.ppc
    python3-six-1.13.0-1.noarch
    python3-test-3.7.15-2.ppc
    python3-tools-3.7.15-2.ppc
    python3.9-3.9.17-1.ppc
    python3.9-cffi-1.13.2-2.ppc
    python3.9-cryptography-3.4.7-4.ppc
    python3.9-jinja2-3.0.3-1.noarch
    python3.9-markupsafe-2.1.2-1.ppc
    python3.9-packaging-19.2-2.noarch
    python3.9-pycparser-2.19-2.noarch
    python3.9-pyparsing-2.4.4-2.noarch
    python3.9-pyyaml-5.4.1.1-2.ppc
    python3.9-resolvelib-0.5.4-1.noarch
    python3.9-six-1.13.0-2.noarch
    rpm-python3-4.15.1-32_1.ppc
    

    $PATH includes the following part:

    :/opt/freeware/bin:



    ------------------------------
    Niklas
    System Engineer UNIX and Linux on Power
    ------------------------------



  • 8.  RE: Issue When Installing lxml for Python3 using pip

    IBM Champion
    Posted Thu October 05, 2023 03:57 AM

    dnf install libxml2-devel libxslt-devel should solve your problem.



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 9.  RE: Issue When Installing lxml for Python3 using pip

    Posted Thu October 05, 2023 04:23 AM

    Great install of lxml worked, after installing the 2 packages and gcc.

    $ python3.7 -m pip install lxml-4.9.3.tar.gz
    Processing ./lxml-4.9.3.tar.gz
      Preparing metadata (setup.py) ... done
    Using legacy 'setup.py install' for lxml, since package 'wheel' is not installed.
    Installing collected packages: lxml
      Attempting uninstall: lxml
        Found existing installation: lxml 4.9.3
        Uninstalling lxml-4.9.3:
          Successfully uninstalled lxml-4.9.3
      Running setup.py install for lxml ... -                                                                                                                                                   done
    Successfully installed lxml-4.9.3

    Unfortunately Ansible still does not find the module. Any idea how we can solve this?

    ansible-playbook [core 2.14.2]
      config file = /scratch/nv/ansible/hmc/trunk/ansible.cfg
      configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
      ansible python module location = /opt/freeware/lib/python3.9/site-packages/ansible
      ansible collection location = /scratch/nv/ansible/hmc/trunk
      executable location = /opt/freeware/bin/ansible-playbook
      python version = 3.9.17 (main, Jul  5 2023, 06:17:22) [GCC 10.3.0] (/opt/freeware/bin/python3.9)
      jinja version = 3.0.3
      libyaml = True
    ...
        raise Error("Missing prerequisite lxml package. Hint pip install lxml")


    ------------------------------
    Niklas
    System Engineer UNIX and Linux on Power
    ------------------------------



  • 10.  RE: Issue When Installing lxml for Python3 using pip

    IBM Champion
    Posted Thu October 05, 2023 04:26 AM

    Which AIX version do you have?

    How did you install lxml?



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 11.  RE: Issue When Installing lxml for Python3 using pip

    IBM Champion
    Posted Thu October 05, 2023 04:29 AM

    Sorry, I didn't see your output above from pip installation. You used python3.7 for lxml installation, but Ansible uses python3.9:

    $ python3.7 -m pip install lxml-4.9.3.tar.gz

      python version = 3.9.17 (main, Jul  5 2023, 06:17:22) [GCC 10.3.0] (/opt/freeware/bin/python3.9)

    You must repeat the procedure using python 3.9



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 12.  RE: Issue When Installing lxml for Python3 using pip

    Posted Thu October 05, 2023 04:58 AM

    Oops, thanks for the hint! After installing pip and setuptools for python3.9 the gcc compile failed.

    $ python3.9 -m pip install lxml-4.9.3.tar.gz
    Processing ./lxml-4.9.3.tar.gz
      Preparing metadata (setup.py) ... done
    ...
      gcc -maix64 -fPIC -pthread -O2 -Wno-unused-result -Wsign-compare -DNDEBUG -g -O2 -Wall -I/opt/freeware/include -I/usr/include -DAIX_GENUINE_CPLUSCPLUS -D_ALL_SOURCE -D_LINUX_SOURCE_COMPAT -I/opt/freeware/include -I/usr/include -DAIX_GENUINE_CPLUSCPLUS -D_ALL_SOURCE -D_LINUX_SOURCE_COMPAT -DCYTHON_CLINE_IN_TRACEBACK=0 -I/opt/freeware/include/libxml2 -I/opt/freeware/include -Isrc -Isrc/lxml/includes -I/opt/freeware/include/python3.9 -c src/lxml/etree.c -o build/temp.aix-7205-2135-64-cpython-39/src/lxml/etree.o -w
      src/lxml/etree.c:96:10: fatal error: Python.h: No such file or directory
         96 | #include "Python.h"
            |          ^~~~~~~~~~
      compilation terminated.
      Compile failed: command '/opt/freeware/bin/gcc' failed with exit code 1
      creating tmp
      cc -I/opt/freeware/include/libxml2 -I/opt/freeware/include -I/usr/include/libxml2 -c /tmp/xmlXPathInit0fivugdw.c -o tmp/xmlXPathInit0fivugdw.o
      error: command '/opt/freeware/bin/gcc' failed with exit code 1
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      *********************************************************************************
    

    Fix for this issue is to install: python3.9-devel

    # Python.h is part of python3.9-devel
    $ rpm -qf /opt/freeware/include/python3.9/Python.h
    python3.9-devel-3.9.16-2.ppc

    Our first successful run of the Ansible HMC modules. :-)

    PLAY RECAP ***********************************************************************
    hmc00001     : ok=3 

    Many thanks for your support - you have brought us a big step forward. :-)



    ------------------------------
    Niklas
    System Engineer UNIX and Linux on Power
    ------------------------------