z/OS Platform Evaluation and Test (zPET) runs customer-like workloads in a Parallel Sysplex environment to perform the final verification of new IBM Z hardware and software. For more information about zPET, please check out our community.
Overview
As usage of Python continues to grow in the mainframe environment, the language has become the go-to for new projects, automation, and scripts. While IBM Z Open Automation Utilities (ZOAU) provides many interfaces into z/OS and its resources, a recent zPET team project required – in part – gathering ISPF statistics for members in various datasets. This wasn’t functionality that (at the time of writing) ZOAU had available. After considering various methods of getting this information (such as LMMLIST in batch, DSFS, and more), we decided to leverage ZOAU and IBM Data Set Commander for z/OS.
Data Set Commander (DSC) is a software product that provides enhanced productivity in both ISPF and batch. It includes an interactive menu-driven front-end to ISPF as well as a batch utility with enhanced partitioned data set functions. The batch utility shipped with Data Set Commander is called IQIBUTIL and provides, among other things, a way to easily gather ISPF statistics for dataset members through batch. This capability, along with ZOAU’s ability to run MVS programs, provides a way to retrieve and work with ISPF statistics in Python.
In this blog, we discuss how we get dataset information with ZOAU’s datasets module, how to build the proper statements to run IQIBUTIL with ZOAU’s mvscmd module, and how we combined these sources of data into one usable Python object.
Please note: The snippets and examples showcased are provided as-is and have been slightly simplified for readability and consumability.
Retrieving Dataset Information
The datasets module provides numerous functions for working with z/OS datasets, including the ability to create, modify, and delete datasets and members. Since our project was one of discovery, we only needed to retrieve information for certain datasets, and this led us to the datasets.listing() function.
The datasets.listing() function retrieves various dataset information, such as the record format, block size, used space, and more. Here is a simple example using this function:
from zoautil_py import datasets
dsn_name = "ZPET.DSN001"
my_dataset = datasets.listing(dsn_name)[0]
print(f"Object type:\n{type(my_dataset)}")
print()
print(f"Object contents:\n{my_dataset.__dict__}")
Output from the above example:
Object type:
<class 'zoautil_py.types.Dataset'>
Object contents:
{'name': 'ZPET.DSN001', 'recfm': 'FB', 'lrecl': '80', 'dsorg': 'PO', 'volume': VOLXYZ, 'block_size': '3120', 'last_referenced': '2023/10/02', 'used_space': '24516248', 'total_space': '27198720', 'vrbs': 0}
To iterate over a list of datasets, extracting volume details (though other information could also be accessed), and saving that in a dictionary for future use, we used the following:
from zoautil_py import datasets
# List of datasets needed to process
my_dsns = ["ZOS.CNTL.JCL",
"ZPET.PLEX1.REXX",
"USERA.LOCAL.JOBS",
"ZPET.DSN001"
]
# Dictionary to store final results
member_info = dict()
# Iterate over each dataset
for dsn in my_dsns:
listing_info = datasets.listing(dsn)[0]
member_info[dsn] = {"volume": listing_info.volume}
To verify we have the volume information for each dataset, we can loop over the member_info dictionary, printing out the details. Here is how we printed out the dataset and volume information:
# Print out the contents of the dictionary so far
for dsn, dsn_info in member_info.items():
print(f"Dataset: {dsn}, volume: {dsn_info.get('volume')}")
Example output:
Dataset: ZPET.DSN001, volume: VOLXYZ
Dataset: ZPET.PLEX1.JOBS, volume: PAKAAA
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1
Dataset: ZOS.CNTL.JCL, volume: VOL01C
Running the IQIBUTIL program with JCL
The Data Set Commander’s IQIBUTIL program provides a batch utility for replacing IEBCOPY with enhanced partitioned data set manipulation functions. In addition to standard IEBCOPY functions, with IQIBUTIL you can list, copy, rename, delete, and restore members, with or without filters. In our case, we are interested in pulling the ISPF statistics for all members in a dataset without any filters. To do this, we needed to use the DIRLIST function of IQIBUTIL. Data Set Commander also provides an interface – the IQIBUTIL Batch Job Wizard - to help generate JCL.
Below is sample JCL for using IQIBUTIL to get the ISPF statistics for all members in the ZOS.CNTL.JCL dataset:
//IQIBUTIL JOB
//IQIBUTIL EXEC PGM=IQIBUTIL,
// PARM='GENS=NONE,NOSTAMP'
//STEPLIB DD DISP=SHR,DSN=DSC.V9R1M0.SIQILOAD
//IQIBUDFL DD DISP=SHR,DSN=DSC.V9R1M0.SIQIPLIB
//SYSPRINT DD SYSOUT=*
//SYSUT1 DD DISP=SHR,DSN=ZOS.CNTL.JCL
//SYSUT2 DD SYSOUT=*
//SYSIN DD *
DIRLIST OUTDD=SYSUT2,
INDD=SYSUT1,
LIST=YES
SELECT MEMBER=*
//*
The output for the above JCL will be written to the job’s SYSUT2 dataset:
AMBLIST 001 01.02 2018/06/01 2023/01/26 16:12 7 7 0 USER01
COPYDS 001 01.02 2023/06/28 2023/06/28 14:22 15 12 0 USER01
DFDSSPRT 001 01.01 2018/02/06 2018/02/06 08:33 16 13 0 USER02
DFDSSPR1 001
DLTVSAM 001 01.01 2018/03/05 2018/03/05 09:07 8 7 0 USER02
DSSPHYS 001 01.01 2019/04/17 2019/04/17 09:49 16 13 0 SYSPRG1
DSSPRINT 001 01.00 2018/02/01 2018/02/01 09:42 10 10 0 SYSPRG2
DUMPVTOC 001 01.00 2018/05/16 2018/05/16 16:50 9 9 0 ZOSUSER
ICETOOL 001 01.00 2023/08/16 2023/08/16 10:10 9 9 0 USER01
IEBPDSE 001
LISTVTOC 001 01.01 2019/04/12 2019/04/12 09:58 10 12 0 USER01
Running the IQIBUTIL program with Python and ZOAU
While the above JCL is fine for many scenarios, if we needed to run IQIBUTIL as part of a larger Python program, we can leverage ZOAU and its mvscmd module. The mvscmd.execute() function allows us to run an MVS program, and we can use this to create a Python version of our above JCL. We will also need the import ZOAU’s DDStatement and DatasetDefinition.
The first step we need to do is create a temporary dataset with the contents we will provide to IQIBUTIL’s SYSIN. We found that this method is the best approach for handling in-line content with ZOAU. Here is an example of creating the temporary dataset:
from zoautil_py import datasets
temp_sysin_dsn_name = "LOCAL.IQIBUTIL.SYSIN.TEMP"
datasets.create(temp_sysin_dsn_name, type="SEQ", record_length=80,
record_format="FB", block_size="27920", primary_space="5M")
datasets.write(temp_sysin_dsn_name, content="""
DIRLIST OUTDD=SYSUT2,
INDD=SYSUT1,
LIST=YES
SELECT MEMBER=*
""")
With the SYSIN dataset created, we can prepare the dataset definition (DD) statements and run the IQIBUTIL program. We construct a list of these DD statements, and we leverage the DDStatement and DatasetDefinition classes provided by ZOAU to accomplish this. Here is the Python equivalent of our above JCL:
from zoautil_py import mvscmd
from zoautil_py.types import DDStatement, DatasetDefinition
dsn_name = "ZOS.CNTL.JCL"
temp_sysin_dsn_name = "LOCAL.IQIBUTIL.SYSIN.TEMP"
# Create the DD statements used by IQIBUTIL
dd_statements = []
dd_statements.append(DDStatement("STEPLIB",
DatasetDefinition('DSC.V9R1M0.SIQILOAD')))
dd_statements.append(DDStatement("IQIBUDFL",
DatasetDefinition('DSC.V9R1M0.SIQIPLIB')))
dd_statements.append(DDStatement("SYSPRINT", '*'))
dd_statements.append(DDStatement("SYSUT1", DatasetDefinition(dsn_name)))
dd_statements.append(DDStatement("SYSUT2", '*'))
dd_statements.append(DDStatement("SYSIN",
DatasetDefinition(temp_sysin_dsn_name)))
# Run the IQIBUTIL program
response = mvscmd.execute(pgm="IQIBUTIL",
pgm_args="GENS=NONE,NOSTAMP",
dds=dd_statements)
# Convert the response to a dictionary
return_dictionary = response.to_dict()
# Print the command’s output
print(return_dictionary.get("stdout_response"))
Output:
AMBLIST 001 01.02 2018/06/01 2023/01/26 16:12 7 7 0 USER01
COPYDS 001 01.02 2023/06/28 2023/06/28 14:22 15 12 0 USER01
DFDSSPRT 001 01.01 2018/02/06 2018/02/06 08:33 16 13 0 USER02
DFDSSPR1 001
DLTVSAM 001 01.01 2018/03/05 2018/03/05 09:07 8 7 0 USER02
DSSPHYS 001 01.01 2019/04/17 2019/04/17 09:49 16 13 0 SYSPRG1
DSSPRINT 001 01.00 2018/02/01 2018/02/01 09:42 10 10 0 SYSPGR2
DUMPVTOC 001 01.00 2018/05/16 2018/05/16 16:50 9 9 0 ZOSUSER
ICETOOL 001 01.00 2023/08/16 2023/08/16 10:10 9 9 0 USER01
IEBPDSE
LISTVTOC 001 01.01 2019/04/12 2019/04/12 09:58 10 12 0 USER01
...
With this done, we can then delete our temporary SYSIN dataset:
from zoautil_py import datasets
temp_sysin_dsn_name = "LOCAL.IQIBUTIL.SYSIN.TEMP"
datasets.delete(temp_sysin_dsn_name)
Combining Information
The last step is to merge the original dataset information with the member statistics. We will combine this information into a single Python dictionary, utilizing the member_info variable we created earlier. We will also rewrite our IQIBUTIL Python code to loop over the list of datasets stored in the my_dsns variable. Finally, we will parse the output of IQIBUTIL to identify the keys and values for each member statistic. The entire Python script now looks like:
from zoautil_py import datasets, mvscmd
from zoautil_py.types import DDStatement, DatasetDefinition
# List of datasets needed to process
my_dsns = ["ZPET.DSN001",
"ZPET.PLEX1.REXX ",
"USERA.LOCAL.JOBS",
"ZOS.CNTL.JCL"
]
# Name of the temporary data SYSIN dataset
temp_sysin_dsn_name = "LOCAL.IQIBUTIL.SYSIN.TEMP"
# Dictionary to store final results
member_info = dict()
# Create temporary SYSIN dataset
datasets.create(temp_sysin_dsn_name, type="SEQ", record_length=80, record_format="FB",
block_size="27920", primary_space="5M")
datasets.write(temp_sysin_dsn_name, content="""
DIRLIST OUTDD=SYSUT2,
INDD=SYSUT1,
LIST=YES
SELECT MEMBER=*
""")
# Iterate over each dataset
for dsn in my_dsns:
listing_info = datasets.listing(dsn)[0]
member_info[dsn] = {"volume": listing_info.volume,
"members": []}
# Create the DD statements used by IQIBUTIL
dd_statements = []
dd_statements.append(DDStatement("STEPLIB",
DatasetDefinition('DSC.V9R1M0.SIQILOAD')))
dd_statements.append(DDStatement("IQIBUDFL",
DatasetDefinition('DSC.V9R1M0.SIQIPLIB')))
dd_statements.append(DDStatement("SYSPRINT", '*'))
dd_statements.append(DDStatement("SYSUT1", DatasetDefinition(dsn)))
dd_statements.append(DDStatement("SYSUT2", '*'))
dd_statements.append(DDStatement("SYSIN",
DatasetDefinition(temp_sysin_dsn_name)))
# Run the IQIBUTIL program
response = mvscmd.execute(pgm="IQIBUTIL",
pgm_args="GENS=NONE,NOSTAMP",
dds=dd_statements)
# Convert the response to a dictionary
return_dictionary = response.to_dict()
# Process each line of the program's output
for stat_line in return_dictionary.get("stdout_response").split("\n"):
stat_line = stat_line.strip()
# IQIDSCPY indicates the end of the member statistics section
if "IQIDSCPY" in stat_line:
break
# Parse the fields from each line
stats_name = stat_line[:8].strip()
stats_created = stat_line[19:29].strip()
stats_updated = stat_line[30:40].strip()
stats_userid = stat_line[74:84].strip()
# Combine statistic information with the original member_info dictionary
stats_dict = {}
stats_dict["name"] = stats_name
stats_dict["created"] = stats_created
stats_dict["updated"] = stats_updated
stats_dict["userid"] = stats_userid
member_info[dsn]["members"].append(stats_dict)
# Delete temporary SYSIN dataset
datasets.delete(temp_sysin_dsn_name)
# Print out the final contents of the dictionary
for dsn, dsn_info in member_info.items():
for member in dsn_info.get('members'):
member_string = ", ".join([f"{k}: {v}" for k, v in member.items()])
print(f"Dataset: {dsn}, volume: {dsn_info.get('volume')}, {member_string}")
Here is the output from the above example:
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: AMBLIST, created: 2018/06/01, updated: 2023/01/26, userid: USER01
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: COPYDS, created: 2023/06/28, updated: 2023/06/28, userid: USER01
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: DFDSSPRT, created: 2018/02/06, updated: 2018/02/06, userid: USER02
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: DFDSSPRT1, created: , updated: , userid:
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: DLTVSAM, created: 2018/03/05, updated: 2018/03/05, userid: USER02
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: DSSPHYS, created: 2019/04/17, updated: 2019/04/17, userid: SYSPRG1
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: DSSPRINT, created: 2018/02/01, updated: 2018/02/01, userid: SYSPRG2
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: DUMPVTOC, created: 2018/05/16, updated: 2018/05/16, userid: ZOSUSER
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: ICETOOL, created: 2023/08/16, updated: 2023/08/16, userid: USER01
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: IEBPDSE, created: , updated: , userid:
Dataset: ZOS.CNTL.JCL, volume: VOL01C, name: LISTVTOC, created: 2019/04/12, updated: 2019/04/12, userid: USER01
Dataset: ZPET.DSN001, volume: VOLXYZ, name: ZWESLSTC, created: 2022/05/25, updated: 2022/05/31, userid: ZOSUSER
Dataset: ZPET.DSN001, volume: VOLXYZ, name: ZWESVSTC, created: 2020/06/01, updated: 2020/06/01, userid: USER01
Dataset: ZPET.PLEX1.REXX, volume: PAKAAA, name: CHECK, created: 2015/04/30, updated: 2015/05/11, userid: ZOSUSER
Dataset: ZPET.PLEX1.REXX, volume: PAKAAA, name: RUNCMD, created: 2007/04/26, updated: 2007/06/20, userid: ZOSUSER
Dataset: ZPET.PLEX1.REXX, volume: PAKAAA, name: SDSFREXX, created: 2015/04/29, updated: 2015/08/04, userid: SYSPRG1
Dataset: ZPET.PLEX1.REXX, volume: PAKAAA, name: ZFSTEST, created: 2007/09/17, updated: 2008/11/20, userid: ZOSUSER
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: BPXBATCH, created: 2022/10/07, updated: 2022/10/07, userid: USER01
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: JOBCNCL, created: 1996/02/01, updated: 2016/02/19, userid: SYSPRG1
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: STARTALL, created: 1996/02/01, updated: 2010/03/03, userid: SYSPRG1
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: STOPALL, created: 1996/02/01, updated: 2013/07/07, userid: SYSPRG1
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: TESTJB, created: 1996/02/01, updated: 2016/06/10, userid: SYSPRG2
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: VTOCINF, created: 2022/06/15, updated: 2023/08/23, userid: ZOSUSER
Dataset: USERA.LOCAL.JOBS, volume: LOCAL1, name: ZZZUPDN, created: 1996/02/01, updated: 2017/06/09, userid: SYSPRG2
This information and format served our purposes. Now, with the content in a dictionary format, it would be easy to convert to another object (like a DataFrame) or create a local class and object. This data can also be written to a file (such as a CSV, JSON, or others) to be consumed by another process or program.
Note: The above examples are straightforward and without error checking. It is advised to implement Python best practices, logging, and error checking as appropriate.
Conclusion
Overall, we found Python, ZOAU, and Data Set Commander were great tools to use in a task such as ours, and we were impressed how well things worked together. ZOAU’s mvscmd module was a critical piece in this process that allowed us to run Data Set Commander’s IQIBUTIL program easily. As we approach future projects that require processing datasets, we will be sure to consider both ZOAU and Data Set Commander.
References
IBM Z Open Automation Utilities
ZOAU samples repository
IBM Data Set Commander for z/OS
Related Articles
The Journey from JCL to Python: so easy even an old mainframer can do it. by Frank De Gilio
Top 8 Reasons for using Python instead of REXX for z/OS by Frank De Gilio
Authors
Michael Cohoon (mtcohoon@us.ibm.com)
Jean Mothersele (jean.mothersele@ibm.com)