Python is one of the most popular programming languages among developers all around the world. Simplified and user-friendly syntax is one of the reasons why Python is loved.
However, the high scalability of Python is one of the outstanding features that stands out among other languages.
For example, Python can be used for automation or continuous integration tools, and on top of that, it plays an essential role in more complex projects such as AI or data analysis thanks to its rich ecosystem of packages and modules.
As mentioned above, Python can be used for various tasks. Therefore, choosing a suitable method is essential to facilitate collaboration and increase work effectiveness while using Python.
We list 4 scenarios showing common use cases and best practices, including:
- Using the Python interpreter as a requirement for various tools
- Building local sandboxes
- Building customized deployment copies
- Creating customized deployment with specific site-packages
Lots of terms including virtual environment, pip and venv will be used in explaining the above 4 scenarios.
Let me give you a brief explanation for those of you are not familiar with these terms.
Python virtual environment is a Python tool for dependencies management which allows a user to create an isolated environment. Each isolated environment can have different version of Python and a list of installed site-packages. To create the virtual environment, you can utilize a virtual environment module that Python includes, which is venv.
For more detailed information about the virtual environment, please refer to the video How to set up a virtual environment using IBM Open Enterprise Python for z/OS.
Later in this blog, we will discuss available C/C++ compilers and best practices for using them as we support multiple C/C++ compilers with IBM Open Enterprise SDK for Python 3.11.
4 scenarios and best practices
1. Using the Python interpreter as a requirement for various tools
In this scenario the Python interpreter is used as prerequisite of the tools such as Ansible or Buildbot.
Our suggestion: Simply setup the environment and use Python from default install. You don’t need to be concerned about other aspects of Python.
2. Building local sandboxes
In this scenario you would like to create your own playground in order to proceed with various projects avoiding pollution of the default Python install.
Our suggestion: Create a virtual environment based off the default Python install to build the local sandboxes.
When you are ready to distribute your packages or application, or you want to store the dependencies for re-installation purposes, you can run "pip freeze" and redirect the output to the requirement.txt.
>>> pip freeze --local > requirements.txt
Use the file to reinstall the packages by running following command.
>>> pip install –r requirements.txt
3. Building customized deployment copies
In this scenario you have multiple machines with the same configuration, an automated way to create machines and deploy the same environment routinely or you would like to make multiple copies of your customized environment for your users.
Building customized deployment copies will be a great option for you to take.
Our suggestion: Unpax the Python install image, and customize the environment and pax it again for the purpose of reusing or redistributing your custom environment.
4. Creating customized deployment with specific site-packages
This scenario will be beneficial when you want to maintain a specific Python environment to be able to manage workspaces for your users in a convenient and secure manner.
Our Suggestion: Use virtual environments and symlinks to customize the environment suitable for your product or project. Therefore, it decreases burden of initial setup for users and decreases the likelihood of violating security regulations.
Moreover, this model will play a huge role when new updates come out. You can simply change what the symlink points at, and automatically move everybody up to the next version without needing them to change anything.
The symlink model is consists of 3 main directories:
python-base, python-deploy-2022 and python-deploy.
- python-base
- Location that you keep the clean install of Python
- python-deploy-2022
- Customized global virtual environment that everyone can access but cannot modify
- python-deploy
- The symlink which points to specific virtual environment you want your user’s workload to use
Now, let’s go through the following step-by-step process of using the symlink model.
- Create a global virtual environment.
>>> python3 -m venv python-deploy-2022 –system-site-packages
>>> ls
python-base python-deploy-2022
- Customize the environment by installing required packages
Note that we name the virtual environment python-deploy-2022, however, if you change the name here, you need to change that name in the rest of the instructions as well.
In this example, Django is installed for setting up the environment, any other packages that you required can be installed in a similar manner.
>>> . ./python-deploy-2022/bin/activate
>>> pip3 install Django
Collecting Django
Downloading Django-4.0.6-py3-none-any.whl (8.0 MB)
|████████████████████████████████| 8.0 MB 5.2 MB/s
Requirement already satisfied: asgiref<4,>=3.4.1 in ./python-deploy-2022/lib/python3.9/site-packages (from Django) (3.5.2)
Requirement already satisfied: sqlparse>=0.2.2 in ./python-deploy-2022/lib/python3.9/site-packages (from Django) (0.4.2)
Installing collected packages: Django
Successfully installed Django-4.0.6
- Change the permission
Ensure proper permission and ownership in python-deploy-2022.
In our example, we set the virtual environment to 755 so that we can open the virtual environment for everyone to use but not to modify.
>>> chmod -R 755 python-deploy-2022
- Create the symlink
We create the symlink of python_deploy that points to python-deploy-2022.
>>> ln -sf python-deploy-2022 python-deploy
>>> ls -al
drwx------ 3 CDEV 8192 Jun 22 13:24 python-base
lrwxrwxrwx 1 CDEV 19 Jun 21 13:39 python-deploy -> python-deploy-2022/
drwxr-xr-x 5 CDEV 8192 Jun 21 13:35 python-deploy-2022
- Activate the virtual environment and try it out
Now activate python-deploy.
>>> . ./python-deploy/bin/activate
>>> pip3 list
Package Version
------------------ -------
asgiref 3.5.2
cffi 1.14.0
click 8.1.3
cryptography 3.3.2
Django 4.0.6
ebcdic 1.1.1
importlib-metadata 4.11.4
itsdangerous 2.1.2
Jinja2 3.1.2
MarkupSafe 2.1.1
numpy 1.18.2
pip 21.1.3
pycparser 2.20
setuptools 57.0.0
six 1.16.0
sqlparse 0.4.2
Werkzeug 2.1.2
zipp 3.8.0
zos-util 1.0.0
As you can see, even though the user did not install any packages, Django and other packages are already there.
To learn more about IBM Open Enterprise SDK for Python, visit IBM Open Enterprise SDK for Python.
Best Practices for using IBM C/C++ Compilers with IBM Open Enterprise SDK for Python3.11
C/C++ compilers available on z/OS
There are multiple compilers that you can use with IBM Open Enterprise SDK for Python 3.11. It is recommended to use either IBM C/C++ For Open Enterprise Languages on z/OS 2.0 or use IBM Open XL C/C++ 1.1 for z/OS 2.4 and z/OS 2.5 because they provide the highest compatibility with PyPI packages.
The following list provides the download address for each C/C++ compiler.
Build packages with distutils or setuptools
To build packages with distutils or setuptools with a given compiler, set the following environment variables:
export CC=<path_to_c_compiler>
export CXX=<path_to_cxx_compiler>
export LDSHARED=<path_to_c_compiler>
If the CC, CXX, and LDSHARED variables are not set, when you build a package with distutils or setuptools, they will automatically attempt to discover which compiler to use by searching each compilers default install location, in the same order as listed above. The default order and install locations follow the list below:
- IBM C/C++ For Open Enterprise Languages on z/OS 2.0:
/usr/lpp/IBM/oelcpp/v2r0/bin/clang
- IBM Open XL C/C++ 1.1 for z/OS 2.4 and z/OS 2.5:
/usr/lpp/IBM/cnw/v1r1/openxl/bin/ibm-clang
- IBM XL C/C++ 2.4.1 for z/OS 2.4:
/bin/xlclang
- IBM z/OS XL C/C++:
/bin/xlc
Note: There are binary compatibility issues when mixing compilers, and you should avoid mixing compilers.
Recommended Configuration for IBM z/OS XL C/C++
If you are using IBM z/OS XL C/C++, then it’s required to set the following environment variables before building any packages:
export _CC_CCMODE=1
export _CXX_CCMODE=1
export _C89_CCMODE=1
export _CC_EXTRA_ARGS=1
export _CXX_EXTRA_ARGS=1
export _C89_EXTRA_ARGS=1
Without these environment variables set up, you may encounter the following error while building:
FSUM3010 Specify a file with the correct suffix (.C, .hh, .i, .c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name
Troubleshooting on using IBM z/OS XL C/C++
For more information about troubleshooting C/C++ compilers, see “Chapter 5. Using IBM C/C++ compilers with IBM Open Enterprise SDK for Python 3.11” on page 11 and “Errors for incorrectly tagged files” on page 40 from IBM Open Enterprise SDK for Python 3.11 - User's Guide.