z/OS Connect Enterprise Edition - Group home

Using Splunk with z/OS Connect EE

By Yun Han Li posted Mon August 10, 2020 04:42 AM

  
This blog was originally published on May 22, 2019, by Nigel Williams, IBM Systems Center Montpellier.


Introduction

In this article we provide an overview of how z/OS Connect EE audit data can be viewed in a Splunk dashboard.

Common Data Provider for z Systems

IBM Common Data Provider for z Systems provides the infrastructure for accessing IT operational data from z/OS systems and streaming it, in a consumable format, to an analytics platform such as Elasticsearch, Apache Hadoop or Splunk. It monitors z/OS log data and SMF data and forwards it to the configured destination. Over 140 different data sources are supported including all standard SMF records, Syslog and application logs from CICS, IMS, Netview and other subsystems. A web-based configuration tool is used to specify what data you want to collect from your z/OS system, where you want the data to be sent, and what form you want the data to arrive in at its destination. This configuration information is contained in a policy.

In March 2019, Common Data Provider for z Systems added the ability to consume SMF data (record type 123) written by the z/OS Connect EE audit interceptor. Auditing provides traceability and accountability allowing you to capture and record which APIs have been called by which users. The example below shows an extract of the output from the audit interceptor.

JOBNAME ARRIVAL-DATE ARRIVAL-TIME TARGET-URI SERVICE-OR-API METHOD USER-NAME MOPZCEW 2019/05/14 12:44:31 /catalogManager/v1.0/orders catalog_v1.0 GET User1

See here for a definition of the format of z/OS Connect EE audit records.

Figure 1 shows a Common Data Provider for z Systems policy for streaming z/OS Connect EE audit records (SMF 123 records) to Splunk.

SMF-123-Splunk-data-stream
Figure 1 Streaming z/OS Connect EE audit records to Splunk

The Common Data Provider for z Systems Splunk application (https://splunkbase.splunk.com/app/3822/) demonstrates how to use mainframe operational data that has been streamed to a Splunk environment. It provides Splunk dashboards that show near real-time data from SMF 30 and Syslog data. IBM Z Operations Analytics (https://splunkbase.splunk.com/app/4498/) includes a broader set of IBM Z operational dashboards and insights across a variety of subsystems.

Sample scenario

In this sample scenario, we have created a Splunk dashboard (Figure 2) that provides a view of a z/OS Connect EE API workload.

splunk-may14
Figure 2 Sample Splunk dashboard

Each of the charts in the dashboard are associated with search strings defined in the Splunk user interface.

  • The API Requests chart shows the number of requests per minute for the different APIs of the workload (catalog, customer and phonebook APIs). The search string for this chart is shown below:
  • * sourcetype="zos-smf_123*" | timechart count(SM123SSI) by API_SERVICE_NAME​

    The sourcetype defines the type of data being searched, timechart defines the type of chart, SM123SSI represents the type of subsystem (z/OS Connect EE in this case) and records are counted by the API-SERVICE-NAME which represents the name of the API.

  • The customer API by HTTP method chart shows the different HTTP methods used to invoke the customer API. The search string for this chart is shown below:
* sourcetype="zos-smf_123*" API_SERVICE_NAME="customer*" | stats count by REQ_METHOD
The search strings for the other APIs is the same except for the API_SERVICE_NAME.

  • The catalog API by version chart shows a breakdown of requests for the different versions of the catalog API. The search string for this chart is shown below:
* sourcetype="zos-smf_123*" API_SERVICE_NAME="catalog*" | stats count by API_SERVICE_NAME
This chart allows us to monitor whether older versions of an API are still being used.

  • The catalog API by channel chart shows a breakdown of requests for the catalog API across different channels. The search string for this chart is shown below:
* sourcetype="zos-smf_123*" API_SERVICE_NAME="catalog*" | eval t_channel = case(USER_NAME == "User1", "Call Center", USER_NAME == "User2", "Partner", USER_NAME == "User3", "Web", USER_NAME == "User4", "Mobile")| stats count by t_channel
This chart gives us a view of API requests across the Call Center, Mobile, Partner and Web channels. We use the request user ID to distinguish between the different channels.

Summary

This article shows a simple example of how z/OS Connect EE SMF data can be viewed in a Splunk dashboard. Splunk is capable of ingesting, correlating, and visualizing data from across your entire enterprise so the range of dashboards that you can create is almost limitless.

More information

See the links below for more information on using the Common Data Provider for z Systems with Splunk:

The IBM Common Data Provider for z Systems Dashboards on Splunk
IBM Common Data Provider for z Systems

This article was written by Systems Lab Services Montpellier. Thanks to John Strymecki of the IBM Z Advanced Technology Group for the creation of the Splunk dashboard.

0 comments
22 views