For configuration files that contain only ASCII characters:
chtag -tc ISO8859-1 config.json
chtag -tc ISO8859-1 settings.yaml
chtag -tc ISO8859-1 pyproject.toml
For configuration files that may contain Unicode characters (international text, emoji, etc.) or come from distributed platforms:
chtag -tc UTF-8 config.json
chtag -tc UTF-8 settings.yaml
Detecting file encoding: If you're unsure of a file's encoding, you can use the file command to get information about the file, though it may not always accurately detect the encoding. The most reliable method is to check the file's CCSID (Coded Character Set Identifier) using ls -T:
# Check current file tag
ls -T config.json
You can also use Python's chardet library (if installed) to detect encoding:
python3 -c "import chardet; data=open('config.json','rb').read(); print(chardet.detect(data))"
Best practice: ISO8859-1 is the encoding used by default when Python creates files on z/OS. If you're unsure of the appropriate encoding, ISO8859-1 is a safe default for files that will be processed primarily on z/OS. Use UTF-8 when you need explicit Unicode support or when sharing files with other platforms.
Tagging shell scripts
Recommendation: Tag shell scripts as ISO8859-1.
chtag -tc ISO8859-1 setup.sh
chtag -tc ISO8859-1 deploy.sh
Shell scripts on z/OS typically use EBCDIC encoding. However, if your shell scripts are managed in Git or developed on distributed platforms, they may also use ASCII (ISO8859-1) encoding and should be tagged accordingly.
Auto-conversion (Enhanced vs. Unenhanced ASCII)
The T=on flag enables automatic conversion between encodings:
# Enable auto-conversion (Enhanced ASCII)
chtag -tc ISO8859-1 file.txt
# Disable auto-conversion (Unenhanced ASCII)
chtag -tcu ISO8859-1 file.txt
Use Enhanced ASCII (T=on) for text files that require automatic conversion between encodings when accessed by different applications.
Use Unenhanced ASCII (T=off) for binary data, files where byte-for-byte integrity is critical, and performance-critical applications (avoids conversion overhead).
Python file I/O best practices
Always specify encoding explicitly to avoid platform-dependent behaviour:
with open('data.txt', 'r', encoding='iso8859-1') as f: content = f.read()
Reading ISO8859-1 files
# Standard ISO8859-1 file
with open('script.py', 'r', encoding='iso8859-1') as f:
content = f.read()
Reading UTF-8 files
# UTF-8 file from distributed systems
with open('api_data.json', 'r', encoding='utf-8') as f:
data = json.load(f)
Reading IBM-1047 files
# EBCDIC IBM-1047 file
with open('ebcdic_file.txt', 'r', encoding='ibm1047') as f:
content = f.read()
Writing ISO8859-1 files
# Write ISO8859-1 and tag the file
import os
filename = 'output.txt'
with open(filename, 'w', encoding='iso8859-1') as f:
f.write('Processing complete\n')
Writing UTF-8 files
# Write UTF-8 with Unicode characters and tag the file
import os
filename = 'output.txt'
with open(filename, 'w', encoding='utf-8') as f:
f.write('Hello, World! \n')
Writing IBM-1047 files
# Write IBM-1047 (EBCDIC) file
import os
filename = 'output.txt'
with open(filename, 'w', encoding='ibm1047') as f:
f.write('Processing complete\n')
Binary files
# Binary mode - no encoding conversion
with open('data.bin', 'rb') as f:
binary_data = f.read()
with open('output.bin', 'wb') as f:
f.write(binary_data)
Environment Variables
Controls automatic conversion for untagged files:
# Enable auto-conversion (default: ON)
export _BPXK_AUTOCVT=ON
# Disable auto-conversion (strict mode)
export _BPXK_AUTOCVT=OFF
_BPXK_AUTOCVT=ON is required for using Python on z/OS. It enables automatic conversion between encodings for untagged files.
Important: It is not recommended to set _BPXK_AUTOCVT in your ~/.profile or ~/.bashrc as it has a performance impact if you enable it and run various applications. Instead, set it only when running Python
# Set only when running Python
export _BPXK_AUTOCVT=ON python3 script.py
For production environments, consider _BPXK_AUTOCVT=OFF to enforce proper tagging discipline, but note that Python requires it to be ON for proper operation.
Note: Python internally sets the _TAG_REDIR_IN, _TAG_REDIR_OUT, and _TAG_REDIR_ERR variables, so you do not need to set them manually.
Common issues and solutions
UnicodeDecodeError
Error:
UnicodeDecodeError: 'iso8859-1' codec can't decode byte 0x85 in position 10
Cause: File is being read as ISO8859-1, but the content within it is not in that encoding (likely EBCDIC).
Solution:
# Check file tag first
import subprocess
result = subprocess.run(['ls', '-T', 'file.txt'], capture_output=True, text=True)
print(result.stdout)
# Read with correct encoding
with open('file.txt', 'r', encoding='ibm1047') as f:
content = f.read()
Note: The file tag indicates what encoding the file claims to be, but the actual content may be in a different encoding. Always verify that the file's tag matches its actual content encoding.
output
Symptom: Text appears as strange characters.
Cause: Encoding mismatch between writer and reader.
# Ensure consistent encoding throughout pipeline
# Writer:
with open('output.txt', 'w', encoding='iso8859-1') as f:
f.write('Processing complete\n')
# Reader:
with open('output.txt', 'r', encoding='iso8859-1') as f:
print(f.read())
Mixing ISO8859-1 and UTF-8 files
Symptom: Some files work, others fail with encoding errors.
Cause: Inconsistent tagging across project files.
Solution:
# Standardize on ISO8859-1 for Python files
find /path/to/project -name "*.py" -exec chtag -tc ISO8859-1 {} \;
# Use UTF-8 only for files with explicit Unicode content
chtag -tc UTF-8 /path/to/unicode_data.json
Quick reference
Python encoding patterns:
# ISO8859-1 read (IBM Python standard)
with open('file.txt', 'r', encoding='iso8859-1') as f:
data = f.read()
# UTF-8 read (for Unicode content)
with open('file.txt', 'r', encoding='utf-8') as f:
data = f.read()
# EBCDIC read
with open('file.txt', 'r', encoding='ibm1047') as f:
data = f.read()
# Binary (no encoding)
with open('file.bin', 'rb') as f:
data = f.read()
Want to learn more about using Python on z/OS? Visit our Python Portal for a convenient way to access more resources.