Using Pandas with IBM Open Enterprise SDK for Python
Pandas is a free and open source Python package used for data manipulation and analysis, being the most widely used analysis tool available for Python. It is fast, flexible, easy to use, and it can be run with IBM Open Enterprise SDK for Python, enabling users to take input directly from a multitude of different sources such as formatted files (.csv, .json, etc…), or through RESTful API calls to external services.
If you have IBM Open Enterprise SDK for Python 3.11, the quickest installation method is to use Python AI Toolkit for IBM z/OS. This is a repository of prebuilt Python packages relating to AI - the benefits of this are that you do not need to have a C or C++ compiler to install these packages, and the installation is significantly quicker. To install Pandas using the Python AI Toolkit for IBM z/OS, you can follow these steps:
- Verify that your Python environment is setup correctly
- (Optional) Service & Support for the packages from the Python AI Toolkit are optional, see the Python AI Toolkit for IBM z/OS for details on how to acquire this
- (Optional) Create a virtual environment
- python3 –m venv venv
- source ./venv/bin/activate
- Install Pandas using the Python AI Toolkit for IBM z/OS
- pip3 install pandas --index-url <Python AI Toolkit url> --trusted-host <Python AI Toolkit url>
If you are on IBM Open Enterprise SDK for Python 3.10 or lower, then there is a fork of Pandas v1.1.2 compatible with z/OS. The following instructions can be used to install Pandas:
- Follow/verify that your Python environment is setup correctly
- All Python environment variables, PATH, LIBPATH, _BPXK_AUTOCVT, _CEE_RUNOPTS, _TAG_REDIR_ERR, _TAG_REDIR_IN and _TAG_REDIR_OUT have been set
- That the CC and CXX environment variables have been set to the appropriate compiler, being the path to either xlc/xlc++ or xlclang/xlclang++
- That the Compiler control variables have been set:
- export _CC_CCMODE=1
- export _CXX_CCMODE=1
- export _C89_CCMODE=1
- export _CC_EXTRA_ARGS=1
- export _CXX_EXTRA_ARGS=1
- export _C89_EXTRA_ARGS=1
- SSH Keys have been setup to allow cloning from Github
- Create a virtualenv to install Pandas into, and activate that venv
- python3 -m venv venv --system-site-packages
- source ./venv/bin/activate
- Install Pandas into the venv using the source from Github
- ./venv/bin/pip3 install git+ssh://git@github.com/pitmanst/pandas.git@v1.1.2.zos
- Fix the permissions following the installation
- find venv/lib/python3.8/site-packages/pandas -name '*.so' -exec chmod 755 {} \;
- find venv/lib/python3.8/site-packages/pandas -name '*.so' -exec chtag -r {} \;
- find venv/lib/python3.8/site-packages/pandas -name '*.x' -exec chtag -r {} \;
If installing into the default Python directory instead of a virtualenv, the following commands can be used instead:
pip3 install setuptools wheel 'Cython>=0.29.16,<3'
pip3 install --no-build-isolation git+ssh://git@github.com/pitmanst/pandas.git@v1.1.2.zos
Followed by the commands in step 4 modified to point to your Python install directory.
Running a Sample Pandas program
$ python3
Python 3.8.5 (heads/pyz_dev-3.8:c68ff1320c, Aug 22 2020, 03:07:56) on zos
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.read_csv("./test.csv")
col1 col2 col3
0 a b c
1 d e f
>>> s = pd.Series([1,2,3])
>>> print(s)
0 1
1 2
2 3
dtype: int64
Troubleshooting
When trying to read from a CSV, I get an error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte
To fix this, make sure to tag your file with the correct encoding. See the IBM Open Enterprise Python for z/OS Troubleshooting page for more information.
Tutorials
For getting started with Pandas, the Pandas website has many tutorials which can assist with learning how to use it.