IBM Open Data Analytics for z/OS (IzODA) is an IBM product aimed at utilizing existing enterprise data to gain analytic insights for businesses. It contains Apache Spark, Anaconda and Optimized Data Layer. Click here to go to our previous Blog where we provided an introduction of IzODA.
The Optimized Data Layer, also known as the IBM Mainframe Data Service for Apache Spark (MDS, or MDSS), acts as a data abstraction layer that offers a common interface to many forms of z/OS data including DB2, VSAM, and SMF. As for SMF data, MDS has new support to stream SMF data in real time when it is configured to access the SMF real-time interface on z/OS. The requester can then use standard SQL SELECT statements to stream SMF records in real time, directly from the SMF in-memory resource. The connection to the SMF in-memory resource is made at the time of the request, and the SQL statement does not reach end of data until the MDS server is stopped or the request is canceled. See Configuring access to SMF data for IT Operational Analytics and Configuring access to System Management Facility (SMF) files for details on how to configure SMF and SMF real-time access with MDS.
On one of our LPARs where the MDS server is running, we were able to stream SMF records real-time from MDS. Here’s how we did it.
First, we already had the following SMF in-memory resource defined in our SMFPRMxx parmlib member:
INMEM(IFASMF.Q.&SYSNAME..INMEM,RESSIZMAX(2G),TYPE(14,15,30,42,60,62,64,80,82,92,100,101,102,110,115,119))
Next, on our MDS server, we enabled real-time access to this SMF in-memory resource by adding the following statements to the localhlq.SAZKEXEC(AZKSIN00) configuration initialization member.
- We configured a local variable in our AZKSIN00 to match the &SYSNAME system symbol that is picked up by our SMFPRMxx when configuring the target SMF in memory resource. We added the statement to define this symbol as a GLOBAL PRODUCT OPTIONS statement as follows:
RTISYS = MVSVAR('SYSNAME')
On the system where we defined this configuration, &SYSNAME resolves to JA0
2. We then used this RTISYS variable value in the ASKZIN00 member when DEFINEing access to the SMF data with the new STREAM option as follows:
"DEFINE SMF NAME(IFASMF.Q."||RTISYS||".INMEM)",
"STREAM(IFASMF.Q."||RTISYS||".STREAM)",
"BUFSIZE(500)",
"TIME(0)"
3. MDS ships with a long list of pre-defined vitual table mappings for all types of SMF records and you can read more about how to enable this with
Configuring access to SMF data for IT Operational Analytics When you access the SMF data mapped by those virtual tables, for example with an application that issues a SQL statement, you also need to pass the data set name as part of of the table name using the format
TableMapName__Data_Set_Namewhere Data_Set_Name is prefixed by two underscores (
__) and the periods in the data set name are replaced with single underscores (
_).
In our SMF Stream example, the Data_Set_Name resolves to
IFASMF_Q_RTISYS_STREAM.
There are two ways to access the SMF records real time with IzODA:
- JDBC for Java and Scala applications - usually run from the Spark environment.
- dsdbc for Python applications running in the Anaconda environment.
We'll use SMF Type 101 records, for example, to show how to access SMF data real time by Scala JDBC connection and Python dsdbc connection.
1. JDBC for Java and Scala applications: Connect through JDBC to the target MDS server, execute the query with executeQuery function to get data from SMF defined stream, iterate and print the result set.
Here is sample scala code to demonstrate this JDBC approach:
import com.rs.jdbc.dv.DvDriverimport java.sql.{Connection, DriverManager, ResultSet}//Set JDBC connection statement, url is the target MDS server urlval driverClassName = "com.rs.jdbc.dv.DvDriver" var pwd = "<pwd>"var usr = "<userName>"val url = "jdbc:rs:dv://x.xxx.xx.xx:8888;DBTY=DVS;CompressionType=UNCOMPRESSED;logconfiguration=/logfile.xml"val conn_str = url + ";user=" + usr + ";password=" + pwd//Define connect fuctiondef connect(conn_str: String): Connection = {//Create a new instance for target driverClass.forName(driverClassName).newInstance()return DriverManager.getConnection(conn_str)}//Define close connection functiondef closeConn(conn:Connection): Unit ={conn.close()}//Create connection with driverval conn = connect(conn_str)//Set the ResultSet attribute to be Forward Only and Read Onlyval statement = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)//Execute Query, SMF_10100_QWAC is the predefined MDS virtual table name for SMF type 101 record //IFASMF_Q_JA0_STREAM is the SMF stream name we defined in our MDS serverval rs = statement.executeQuery("select * from SMF_10100_QWAC__IFASMF_Q_JA0_STREAM")//Iterate Over ResultSet, in this example we only print column "QWACBSC"(Transaction begin time) and "QWACBJST" (Class2 CPU time)while (rs.next) {val name = rs.getString("QWACBSC")+' '+rs.getString("QWACBJST")println(name)//You can add close connection condition stop_condition to stop the Iterateif (stop_condition) {closeConn(conn)}}The SMF Type 101 records will be fetched and printed from the time you execute the query to the time the connection closes. Here’s how the output looks from this simplified bit of sample code:
2019-03-07 03:19:57.105141 1253.340807 2019-03-07 03:19:57.186894 67.883067 2019-03-07 03:19:57.246701 45.304413 2019-03-07 03:19:57.234761 1514.044467 2019-03-07 03:19:57.295502 1341.726939 2019-03-07 03:19:57.436141 1560.382617 2019-03-07 03:19:57.485159 310.244607 2019-03-07 03:19:57.495747 1351.987874...... 2. dsdbc for Python applications: Connect to the target server using the dsdbc package. With dsdbc, you can only connect to a local MDS server. Use dsdbc cursors to execute the query function and print the result set.
Here is sample python code to demonstrate this dsdbc approach:
import dsdbc#Get connection with the local target MDS server, AZKX is the server nameconn=dsdbc.connect(SSID="AZKX")#Get cursorcursor=conn.cursor()#Execute Query, we fetch 2 column data QWACBSC,QWACAJST in virtual table SMF_10100_QWAC which is the pre-defined virtual table name for SMF type 101 record#IFASMF_Q_JA0_STREAM is the SMF stream name we defined in our MDS server
cursor.execute("SELECT QWACBSC,QWACAJST FROM SMF_10100_QWAC__IFASMF_Q_JA0_STREAM")#Fetch one row resultset rs=cursor.fetchone()#Iterate Over ResultSetwhile(rs):print(str(rs[0])+" "+str(rs[1]))#Fetch the next rowrs=cursor.fetchone()#You can add close connection condition stop condition to stop the Iterateif (stop_condition):cursor.close() conn.close()
The output will look like the same as the scala example output above.
The sample code shown here is shared as-is with no support implied or provided.
Hopefully it provides some hints for how you could utilize a real time data consumption strategy with your IzODA applications using the SMF Realtime Interface on z/OS.
Authors:
Hui Wang(cdlwhui@cn.ibm.com)
Xin Xin Dong(xxdongbj@cn.ibm.com)