IBM Common Data Provider for z Systems (CDPz) provides z/OS operational data in a consumable format to multiple subscribers. CDPz acts as a single source for both structured and unstructured data, providing a near real-time data feed of z/OS log data and System Management Facilities (SMF) data to your analytics platform. A simple web-based configuration tool in CDPz allows you to configure what z/OS data is streamed to which target systems (e.g. analytics platforms like Splunk and Elastic Stack), this significantly cuts down on the amount of data streamed to the target systems as only the desired data is sent. For example, you could send just Syslog, SMF 30, SMF 100 and SMF 110 records to Splunk to visualize z/OS system and subsystem performance (DB2, CICS) in Splunk dashboards.
CDPz provides a set of sample dashboards for Splunk and Elastic Stack.
CDPz recently released PTFs in the service stream that provide a number of new capabilities that broaden the scenarios in which CDP can be used.
- Enabling customers who use SMF in data set recording mode (using SYS1.MANx, not log stream mode) to leverage CDPz to stream SMF data through a set of new SMF user exits.
- CDPz now supports streaming IMS Log records (accounting, security violation, transaction level statistics, IMS performance analyzer transaction index). The specific set of IMS log records supported are identified in the detailed section below.
- Improved Documentation for configuration of the CDP UI on z/OSMF and the open streaming API.
The remainder of this blog will provide information on these new key capabilities delivered with the CDPz 1Q 2018 PTFs (UA95494, UA95493, UA95492). For general information on CDPz please see http://ibm.biz/CDPzInfo.
Configuring CDPz to stream SMF records
There are various different methods you can configure in CDPz to stream SMF records in near real-time to subscribers (target systems). Instead of just documenting the new capability delivered in the CDPz 1Q 2018 PTF for streaming SMF records, this section will describe all of the ways that you can configure CDPz to stream SMF data, citing advantages and considerations for each.
Options for configuring streaming SMF records in CDPz include the following:
1. SMF in log stream mode with SMF in-memory buffer
For customers running SMF in log stream mode, it is best for CDPz to collect SMF from the SMF in-memory buffer using the SMF real-time interface. This SMF in-memory buffer is not automatically configured with the log stream configuration in SMF, it is an additional configuration step, see the SMF documentation for details. CDPz must be configured to pull data from the SMF in-memory buffer using the SMF real-time interface.
If SMF is configured for log stream mode but the SMF in-memory buffer is not configured, configuring CDP to pull SMF records directly from log stream has a side consequence that when CDPz pulls data from the log stream (at a configurable interval, which is 1 minute by default), it causes a log stream switch. This switch results in a file being generated with the SMF records since the last switch from which CDPz can read the latest SMF records. The undesirable side effect is that you end up with a lot of smaller files (potentially 1 per minute if you leave the default interval of 1 minute), this can be an annoyance to clean up or potentially cause a concern if you drive automation on the log stream switch. It is for this reason that it is strongly suggested that when SMF is run in log stream mode that you also have the SMF in-memory buffer configured and configure CDPz to pull from the SMF in-memory buffer.
2. SMF in data set recording mode (using SYS1.MANx, not log stream mode)
New capability delivered in CDPz 1Q 2018 PTF
For customers who are running SMF in data set recording mode (SYS1.MANx) (and aren’t moving SMF configuration to log stream mode), they can use the new capability delivered in the CDPz 1Q 2018 PTF to collect and stream SMF data through a set of SMF user exits. Note that the SMF records are not pulled from the SYS1.MANx data set, the records are processed and streamed in near-real time through the new SMF user exits provided by CDPz.
When CDPz is configured for collecting SMF records in data set recording mode, the HBOSMFEX user exit is configured at installation exit points IEFU83, IEFU84 and IEFU85 to process SMF records for CDPz. As SMF records are processed in SMF and the appropriate CDPz user exit is called, the CDPz user exit will copy the record to an above-the-bar 64-bit shared storage buffer. The System Data Engine component of CDPz will periodically read the SMF records from the shared storage buffer and process them, performing the configured transforms and passing them off to the Data Streamer component of CDP to send them to the configured subscribers (e.g. Splunk, Elastic Stack, IOAz, etc). The interval at which the System Data Engine reads records from the shared storage buffer is configurable, the default is 1 minute.
Installation and configuration of CDPz to collect SMF records via the user exits is quite simple. Add the required module to the system link pack area (LPA), install the user exits by updating SYS1.PARMLIB and update SMFRPMxx to call the user exits. There are no other dependencies.
3. Batch
As an alternative to CDPz streaming SMF data in near-real time as the SMF records are submitted, the SDE component of CDPz can be run stand-alone in batch mode to read SMF data from a data set and writing it to a file.
CDPz support for IMS as a data source
New capability delivered in CDPz 1Q 2018 PTF
CDPz now supports streaming a set of IMS Log records to subscribers.
The following is the list of IMS log records now supported in CDPz:
- x’07’, x’0A07’, x’08’, x’0A08’ (accounting records)
- x’10’ (security violation records)
- x’56FA (transaction level statistics records)
- x’CA01’, x’CA20’ (IMS Performance Analyzer Transaction Index records)
Support for streaming IMS log records in near-term time through CDP is a valuable capability as many folks create nightly dashboards that provide IMS data to IT, Line-of-Business support and Line-of-Business managers, but the data is a day old and they want to see the data much sooner, on the same day as it is generated. With this new capability to stream IMS log record through CDPz these folks can stream IMS log records to Splunk or other analytics platforms to visualize results that contain IMS data shortly after it is generated.
While it’s transparent to the user of CDPz, except for x’CA01’ and x’CA20’ records, the IMS log records to be processed by CDPz are submitted as the payload in an SMF reserved record to be processed by CDPz. The flow of these IMS records being processed by CDPz is the following. The IMS LOGWRT user exit writes each of the IMS records in the log buffer to SMF in an SMF reserved record (type 127 subtype 1000, to be used only by IBM). Depending on the SMF recording mode, the IMS log records written to SMF is either captured by the SMF user exit (data set recording mode) and copied to the above-the-bar shared storage buffer, or stored in a SMF in-memory resource (log stream mode). The System Data Engine gatherer component of CDPz reads the SMF records from either the high virtual storage area (data set mode) or SMF in-memory resource (log stream mode). Since CDPz knows that this particular SMF record contained an IMS log record in it’s payload and the record has no other value, the record is not written to the SYS1.MANx data set or to the log stream.
The x’CA01’ and x’CA20’ IMS log records are created by IMS Performance Analyzer batch reporting. These records cannot be streamed near real-time and require a manual activity to process them through CDPz. To process these records you run a batch job provided with CDPz to read the IMS Transaction Index file (x’CA01’) and/or IMS Connect Transaction Index file (x’CA20’) created by IMS Performance Analyzer. For each x’CA01’ and x’CA20’ record it writes an SMF reserved record with the IMS transaction index log record as the payload and the System Data Engine component of CDPz processes it the same way as the other IMS records.
To use the IMS log record support in CDPz it’s important to know that you must have configured CDPz to be using either the CDPz SMF user exits or log stream in-memory mode configuration.
CDPz will continue to add support for additional IMS log record types, such as support for IMS user log records. Other considerations for future enhancement of IMS log record support in CDPz include the ability to process IMS log records across all IMS systems from a single CDPz. Also supporting an IMSplex to process complex transactions in a single IMS system as some records are out of order and some are incomplete, these records need to be set aside and processing completed when all information is available. These enhancements will be considered as CDPz looks to support records that fall into those categories.
Improved Documentation
The CDPz 1Q 2018 PTF provides a number of documentation updates, including
- Improved documentation on configuring the CDP UI on z/OSMF
- More complete documentation on the open streaming API
See https://www.ibm.com/support/knowledgecenter/SSGE3R_1.1.0/welcome.html for the updated CDPz documentation.
For additional information on what’s new in the CDPz 1Q 2018 PTF see https://www.ibm.com/support/knowledgecenter/SSGE3R_1.1.0/cdp_v110.html#cdp_v110.
For additional information on IBM Common Data Provider for z Systems see http://ibm.biz/CDPzInfo.