Message Image  

z/OS Connect - Group home

The IBM Common Data Provider for z Systems Dashboards on Splunk

  

The IBM Common Data Provider for z Systems Dashboards on Splunk

IBM® recently released the IBM Common Data Provider for z Systems™ Splunk application that demonstrates how to use mainframe operational data that has been streamed by IBM Common Data Provider for z Systems V1.1.0 from a z/OS-based IT operations environment to a Splunk environment. It enables you to identify, isolate, and resolve problems across your enterprise from a single interface. This application can easily be downloaded and installed from Splunkbase.

To access the IBM Common Data Provider for z Systems Dashboards, see: https://splunkbase.splunk.com/app/3822/

The IBM Common Data Provider for z Systems Splunk application runs on Splunk Enterprise Server and provides Splunk dashboards that show near real-time data from the z/OS system console log (SYSLOG) and SMF 30. The dashboards show subsystem information on performance and message indicators based on IBM CICS Transaction Server, IBM Db2 and IBM MQ subsystems. These dashboards can be used out of the box for immediate value from your operational data or they can be used as a starting place to create your own specific Splunk dashboards.

To install the app, from your Splunk system, go to the Splunk Apps section and “Browse more Apps”. This will take you to a screen where you can search for apps based on keywords among other criteria. Use the keywords “IBM Common Data Provider” and look for the entry “Dashboards – IBM Common Data Provider”. The application can be installed by simply using the Install pushbutton.

To use the application, you must already have the IBM Common Data Provider for z Systems installed on at least one of your z/OS Systems and capable of streaming data to your Splunk server. On the Splunk server or heavy forwarder you must have the Data Receiver running and configured and you must have installed the IBM Common Data Provider z Systems Buffered Data Ingestion app. This app is included with your IBM Common Data Provider for z Systems mainframe installation and must be downloaded from your mainframe installation and installed into your Splunk server environment. See the IBM Common Data Provider for z Systems knowledge center for information on downloading and installing the Buffered Data Ingestion app.

The IBM Common Data Provider for z Systems must be configured to send the desired data to Splunk and for this app you will need to include SMF 30 and Syslog or Operlog data. The IBM Common Data Provider for z Systems is configured using a web based configuration tool. Below, I’ve created a configuration that will be used to feed the IBM Common Data Provider for z Systems dashboards.



Notice that I have defined two data streams, those being the “z/OS SYSLOG” and the “SMF_030”. Each of the streams include a transform to convert the data to UTF-8. The z/OS SYSLOG data is configured with an extra transform to split the data flow such that Splunk is passed individual messages. Finally the data is sent to a single Splunk server.

One thing that is important to note as you’re creating this configuration is the name of the sourcetypes that will be available on Splunk. Using the pen icon to edit the z/OS SYSLOG Data Stream in the above example will display the details for that data stream.

This panel lets you customize some of the parameters for the data stream, though for the z/OS System Console the defaults are fine. The entry for the “Data Source Type” should be noted as it will be used later in Splunk to restrict searches to specific data source types.

The Common Data Provider for z Systems is capable of very complex configurations where different data is being collecting, transformed in a variety of ways and sent to different servers, but the above configuration is what is required to fully utilize the dashboards.

Once you have the IBM Common Data Provider for z Systems running with the above configuration, you can begin using the application dashboards on Splunk. You can start by selecting the IBM Common Data Provider for z Systems application in the Splunk application navigator


This will bring you to the IBM Common Data Provider for z Systems Welcome Dashboard. The Welcome Dashboard provides an overview of the IBM Common Data Provider of z Systems and the dashboards contained in the app. To understand the dashboards, let’s start by closely examining one of the low-level dashboards, I will use CICS as an example. From the Welcome Dashboard, there is a link to the CICS Dashboard. Selecting that link will bring you to the following dashboard



The initial set of graphs display performance metrics, such as CPU Time and IO Time, from the SMF_030 data source type. Scrolling to the bottom of this dashboard you will see graphs based on messages. Let’s take a deeper look at the first graph that displays the Top 5 CICS Total CPU Time. First select the “Edit” button in the upper right corner of the dashboard and then in the upper right graph, select “Edit Search”

That will display the search that is run to populate this graph.

Breaking down the Search string, the first line is what gathers the data from Splunk using the following parameters.

  • ‘cdp_index’. This is actually a macro that expands into ‘index=”zosdex”’ which restricts the search to only looking in the zosdex index for data. If your installation does not use the default index name for the IBM Common Data Provider for z Systems data, simply change ‘cdp_index’ to be index= followed by the name of the index where your mainframe data is being stored.
  • sourcetype=”zOS-SMF_030”. This defines the type of data being searched as noted above when we were configuring the IBM Common Data Provider for z Systems. IBM Common Data Provider for z Systems typically uses standard field names for the different SMF types. The details for the SMF 30 can be found in the z/OS reference manual
  • SMF30STP=2. SMF 30 is a common address space record that it provides details on jobs that are running on the system. The system will generate SMF 30 when jobs start or stop and also can collect data on an interval basis for long running jobs. The SMF30STP field identifies what type of SMF 30 record we want to look for, and in this case we are only collecting the subtype 2 records, which are interval records. For an overview of the different SMF 30 subtypes see: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieag200/rec30.htm
  • (SMF30PGM=EYU* OR SMF30PGM=DFH*) The SMF30PGM field identifies the name of the program we want to look for. CICS applications will start with either the EYU or DFH prefix.
  • CPUTime To obtain the time that this program has spent on the processor for this interval, we can either use the field SMF30CPT or for this graph we will calculate the CPU time as described in the Common address space work section of the z/OS reference manual.
    To calculate the CPU time, the results of the search will be piped into an eval stage that will create a new field, called CPUTime.
    eval CPUTime=(SMF30CSU * 10) / SMF30CPC * SMF30SUS / 16
  • Key Since this data can be across multiple systems, we will create a key to identify the system and job that this CICS is running on. For this graph, the system name and the jobname (SMF30JBN) are joined together with a colon to create a new key field. JobName isn’t always unique, so may not always be the right choice for a identifier, however with CICS it should be acceptable.
    eval key = system + ":" + SMF30JBN
  • Timechart Finally the results of the search and evaluations will be displayed in a timechart. The timechart will display data on 30 minute intervals (span=30). Note that we are collecting SMF 30 data that is created on an interval. If your SMF 30 data is created every 10 minutes, each point on the timechart will represent 3 SMF 30 records. For this graph we are summing the CPUTime field across the 30 minute span for each job identified by our new key field (sum(CPUTime) by key). Since there can be a large number of CICS jobs across the systems, the timechart is restricted to only showing the top 5 jobs. (limit=5). The timechart has the ability to collect all the other jobs into a single line called “Other”, but this chart does not use that capability (useother=f)

The charts based on SYSLOG data are defined in a similar manner. Still in Edit mode, scroll down to the chart on Storage violations.

And bring up the search in edit mode by selecting the edit search button.

The search is very similar to what we were doing with the SMF 30 data in the prior example. It still starts using the ‘cdp_index’ macro to specify the index. This time the sourcetype is zOS-SYSLOG-Console to indicate that we are looking only for messages coming from the z/OS system console log. And then the search is restricted to only those messages that have the word DFHSM0102, which is the ID for the message from CICS indicating Storage Violations. The rest of the search is the same as for the SMF 30. A key is still generated to identify the System and Job that generated the messages and the timechart is used to display the results, although this time the display is simply a count of the messages returned in each 30 minute span.

A lot can be told of the status of your mainframe by simply using the SMF 30 and syslog records. However, this is just the tip of the iceberg of the great variety of data that can be streamed from your mainframe using the IBM Common Data Provider for z Systems and incorporated into your Splunk applications. Hopefully this has given you quick start to understanding the data and how it can be used to manage the operations of your mainframe systems.

Check out the IBM Common Data Provider for z Systems Dashboards at: https://splunkbase.splunk.com/app/3822/

2/1/2018