Open Source Edge for IBM

The Open Source Edge for IBM is a centralized tool designed to help developers keep up with the growing number of open-source packages and ensure they are enabled to run on IBM Power systems.

With this tool, you can:

• Search for key dependencies and evaluate their availability and suitability for IBM Power.

• Access build scripts, source code, or container images if a package version has been ported.

• Contribute build scripts for packages or request new packages to be added.

The tool is community-friendly and welcomes contributions from both IBM and external developers.

👉 Try the Open Source Edge tool and watch a demo

Optimized Python Wheels for IBM Power

IBM is building a curated set of open-source Python packages natively on IBM Power and making them available via a DevPi server as Python wheels. These packages are selected based on ongoing AI projects across the Power ecosystem and provide a solid foundation for Python development on:

• IBM Power9

• IBM Power10

• IBM Power11

Why Optimized Wheels Matter

By leveraging the unique capabilities of POWER architecture, these optimized wheels can significantly enhance the performance and efficiency of your Python applications on IBM Power servers. Benefits include:

• ⚡ Faster data analysis and machine learning workflows

• 🧠 Improved performance for deep learning and generative AI applications

• 🚀 Increased productivity for development teams

Accessing the Wheel Repository

The wheel repository is hosted on a DevPi Server at:

🔗 https://wheels.developerfirst.ibm.com

This repository is currently not accessible via a Web UI. To browse the repository, install the devpi-client and run the following commands.

📌 Note: A detailed list of available wheels and their licenses is also published in a separate blog.

# Specify the location of the IBM optimized wheels for Power
devpi use https://wheels.developerfirst.ibm.com/ppc64le/linux

# get list of packages
devpi list

# Get versions of all the packages
for p in $(devpi list); do devpi list --all $p; done 

Getting Started with Your Application

To take full advantage of the optimized Python wheels for IBM Power, follow these steps to set up your development environment and test package installation.

✅ Recommended Python Versions

Use Python 3.10, 3.11, or 3.12 — these versions have the broadest support for prebuilt packages in the IBM wheel repository.

🛠️ Step 1: Create a Local Virtual Environment

Creating a virtual environment helps isolate dependencies and ensures a clean setup.

python3.12 -m venv venv
source venv/bin/activate

🧪 Example 1: Install a Package from the IBM Optimized Wheel Repository

Let’s install the pillow package using pip, prioritizing the prebuilt Power binaries from IBM’s DevPi server.

#!/bin/bash
# The pip --prefer-binary option prioritizes prebuilt Power binaries over building packages from source distributions, 
# even if a newer source distribution is available
#
# The pip --extra-index-url option specifies the IBM optimized wheel repository as an additional package index where pip should search for packages,
# in addition to the default PyPI (Python Package Index) or any index specified with --index-url.
$ pip install --prefer-binary pillow --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux

Sample Output:

(.venv) [root@4337a3e79a15 /]# pip install --prefer-binary pillow --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux
Looking in indexes: https://pypi.org/simple, https://wheels.developerfirst.ibm.com/ppc64le/linux
Collecting pillow
  Downloading https://wheels.developerfirst.ibm.com/ppc64le/linux/%2Bf/220/8ec631b26f043/pillow-11.2.1-cp312-cp312-linux_ppc64le.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 17.2 MB/s eta 0:00:00
Installing collected packages: pillow
Successfully installed pillow-11.2.1

🎉 Success! You’ve installed a prebuilt, optimized wheel for IBM Power.

🧪 Example 2: Training a LightGBM Model with AI/ML Packages

This example demonstrates how to build a simple machine learning application using LightGBM, scikit-learn, PyArrow, and Pillow on IBM Power. The application trains a model on an Iris-like dataset, generates predictions, and outputs visuals and metadata. You can find the full source code for this example in the pyeco GitHub repository: 🔗lightgbm-pyarrow-example

🔍 Summary of Steps

1. Create a virtual environment using Python 3.12

2. Install required packages from the IBM optimized wheel repository

3. Resolve runtime errors related to native libraries and dependencies

4. Run the application and validate functionality

5. Generate a pinned requirements.txt for reproducibility

🛠️ Step-by-Step Installation

✅ Create a Virtual Environment

python3.12 -m venv .venv
source .venv/bin/activate

📦 Install AI/ML Packages

Each command uses --prefer-binary and --extra-index-url to prioritize IBM's optimized wheels.

# Install LightGBM (pulls in numpy and scipy)
pip install lightgbm==4.5.0 \
  --prefer-binary \
  --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux

# Install PyArrow
pip install pyarrow==18.0.0 \
  --prefer-binary \
  --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux

# Install Pillow
pip install pillow==11.2.1 \
  --prefer-binary \
  --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux

# Install scikit-learn
pip install scikit-learn==1.5.2 \
  --prefer-binary \
  --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux

🧯 Troubleshooting Runtime Errors

⚠️ Error: libopenblas.so.0 not found

ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

Solution: Set the LD_LIBRARY_PATH to include the OpenBLAS library path:

export LD_LIBRARY_PATH=./.venv/lib/python3.12/site-packages/openblas/lib:$LD_LIBRARY_PATH

⚠️ Error: Additional missing libraries: libgfortran.so.5 , libjpeg.so.62

ImportError: libgfortran.so.5: cannot open shared object file: No such file or directory
ImportError: libjpeg.so.62: cannot open shared object file: No such file or directory

Solution: Install and enable GCC 13 toolset ( for libgfortran.so.5 ) and libjpeg-devel

yum install -y gcc-toolset-13
source /opt/rh/gcc-toolset-13/enable

yum install -y libjpeg-devel

⚠️ Dependency Conflicts - Conflicting versions of numpy caused runtime issues.

(.venv) [root@dcfbad9f5448 /]# python lightgbm-pyarrow-example.py

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.3.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Solution: Downgrade numpy to a compatible version:

pip install numpy==1.26.4 \
  --prefer-binary \
  --extra-index-url=https://wheels.developerfirst.ibm.com/ppc64le/linux

✅ Finalizing the Environment

Once your application runs successfully, freeze the environment to capture exact versions:

pip freeze > requirements.txt

Sample requirements.txt:

joblib==1.5.2
lightgbm==4.5.0
numpy==1.26.4
openblas==0.3.29
pillow==11.2.1
pyarrow==18.0.0
scikit-learn==1.5.2
scipy==1.16.0
threadpoolctl==3.6.0

📌 Why Pinning Versions Matters

• Ensures consistent installs across environments

• Improves reproducibility in CI/CD pipelines

• Simplifies debugging and dependency management

• Prevents unexpected changes in transitive dependencies

📚 Additional Examples

For more hands-on demonstrations, visit the examples directory in the pyeco GitHub repository. These examples are designed to help users understand and utilize IBM Power-optimized Python packages effectively.

🔧 What You'll Find

The repository includes:

• Example workflows using AI/ML and data processing packages

• Shell scripts to automate setup and execution

• Python scripts showcasing real-world use cases

▶️ How to Run an Example

Each example directory contains a run-example.sh script that guides you through the setup and execution process.

This script:

• Detects your Linux distribution and installs required system dependencies

• Creates a dedicated Python virtual environment

• Installs all required packages using a pinned requirements.txt

• Verifies the installation and runs the example along with sub-tests to ensure functionality

cd examples/<example-name>
./run-example.sh

📦 Available Packages and Versions

The IBM Optimized Wheel repository at
🔗 wheels.developerfirst.ibm.com/ppc64le/linux
currently includes a curated selection of Python packages and versions built natively for IBM Power.

We’re actively expanding this list based on community feedback and user needs. If you don’t find a required package in the repository:

1. Visit the Open Source Edge (OSE) portal to check the package’s availability status.

2. Look for an existing build script for the package.

3. Update the script to support the version you need.

4. Contribute your changes by submitting a pull request to the OSE build-scripts repository.

We’ll review your contribution and, as time permits, work to make the optimized version available as a wheel.

💬 Feedback and Suggestions

We’d love to hear from you!

• Try out the examples and share how you plan to use these optimized wheels.

• Suggest packages or versions you'd like to see added.

• Help us improve the repository to better support your development needs.

You can:

• Add a comment to this blog post

• Submit feedback or feature requests by opening an Issue in the pyeco GitHub repository

Your input is invaluable to shaping the future of Python development on IBM Power.

✅ Key Takeaways

• IBM Power-optimized Python wheels are available via a DevPi repository, offering performance and compatibility benefits for AI/ML workloads.

• Use Python 3.10–3.12 for best compatibility with the available wheels.

• Create a virtual environment and use --prefer-binary and --extra-index-url to install packages from the IBM wheel repository.

• The Open Source Edge (OSE) tool helps evaluate package availability and encourages community contributions to build scripts.

• Example workflows in the pyeco GitHub repository provide practical guidance for setup and execution.

• Troubleshooting native library issues (e.g., libopenblas.so, libgfortran.so) may require environment variable updates or system-level installations.

• Use pinned versions in your requirements.txt to ensure reproducibility and stability across environments.

• Community feedback is welcome! Suggest packages, report issues, or contribute via GitHub to help grow the ecosystem.

🙌 Acknowledgments

Thanks to Vipul Ajmera - Vipul.Ajmera@ibm.com for the lightgbm example.

Special thanks to the incredible development team behind this initiative:

• Gerrit Huizenga – gerrit@us.ibm.com

• Hiro Miyamoto – miyamotoh@us.ibm.com

• Janani Janakiraman – janani@us.ibm.com

• Priya Seth – sethp@us.ibm.com

• Nikhil Kalbande – nkalband@us.ibm.com

• Shubham Dayma – Shubham.Dayma@ibm.com

and the Python Ecosystem Team – IBM Power