The 2Q 2019 continuous delivery PTF of the IBM Common Data Provider (CDP) contains several improvements to the policy Configuration Tool user interface. The changes make it easier to connect data streams to subscribers, help you understand which streams you should collect, ease configuring subscribers and provide a path to select basic sets of streams based on commonly used z/OS components and subsystems.
Making it easier to connect to subscribers
The biggest change is that users no longer have to determine if the CDP must transform the data. Previously, a transform to transcribe the data to UTF-8 was added as part of the policy. The user had to determine whether the data was to be split or unsplit based on the subscriber. The UI canvas could contain the stream, a transform to transcribe to UTF-8, and a transform to split the data for each data stream in the policy. The subscriber could conceivably subscribe to the steam and any of the transforms, which could be very confusing in the UI, and lead to duplicate or incorrect data in the subscriber.
Now, the transcription to UTF-8 is removed from the policy, and the CDP will determine how the data should be split. The user simply clicks on the “Subscribe” icon for the stream and adds the connection to the subscriber.
The user will still have to choose a splitter for Generic z/OS Job Output, Generic ZFS File, and Generic VSAM Cluster stream types. Because the data is user defined, the CDP cannot determine how the data should be split into individual records. The Configuration Tool provides splitter transforms to split the data on carriage return/line feed (CRLF Splitter) or a fixed length record (FixedLength Splitter).
There are changes in the way you configure the subscriber as well. The protocol now describes the subscriber more completely. You configure the protocol based on your choice of analytics platform (Elastic stack, Splunk or IOA-LA), whether you use either IZOA or CDP applications on the platform, and whether communications are secured by SSL. The checkbox to send the data split or unsplit has been removed. The CDP now determines if the data should be split based on the subscriber protocol.
Making it easier to get started
The CDPz Configuration Tool has new features to make it easier to select the data streams you need. The first thing you will notice is that we have introduced Starter Sets.
A Starter Set is a collection of logs and streams which comprises a default set of data for a z/OS component or subsystem. Selecting the Starter Set automatically selects the streams in the set and gives the user the opportunity to modify the selections. There are starter sets defined for:
- General z/OS System Monitoring:
- Security
- SYSLOG
- SMF 80
- SMF 80 RELOCATE section
- SMF 80 XRELOCATE section
- CICS
- CICS MSGUSR log
- CICS EYULOG
- SMF 110 KPI record
- WebSphere Application Server
- WebSphere SYSLOG
- WebSphere SYSOUT
- SMF 120 subtype 9
- Db2
- SMF 100 subtype 0
- SMF 100 subtype 1
- SMF 100 subtype 2
- SMF 100 subtype 3
- SMF 100 subtype 4
- SMF 100 subtype 5
- IMS
- IMS Termination records
- IMS Start records
- IMS Security violation records
- IMS transaction level statistics
Starter sets are designed to be complete within themselves and independent of each other. That’s the reason SYSLOG is included in more than one Starter Set. The Configuration Tool will detect duplicate streams and resolve them into a single instance on the UI canvas.
The starter sets are intended to provide basic sets of data for a z/OS component. Users often will need to include more extensive data collection. The descriptions for streams have been enhanced to make it clearer what the stream represents, and its place in the SMF record it is derived from.
Filter transforms
The new Configuration Tool has removed the splitter and transcribe transforms for all but generic streams. The Time Filter and REGEX filter transforms are still available.
Building a policy
Creating a policy will be faster and easier than before.
In this example, the policy will collect general z/OS and security metrics for processing by a Splunk server with the IBM Common Data Provider for z Systems Buffered Data Ingestion app installed.
Click on the “Create new policy” icon.
Provide a unique name.
Start by selecting your data streams. Expand “General z/OS system monitoring” and “Security” under “Starter Sets” -> “Common Data Provider for z Systems”. Selecting the top line of the menu will select all streams in the Starter Set
Complete the stream selection by clicking on the “Select” button.
Connect the data streams to the subscriber. Click on the Subscriber icon on any of the streams. Click on the “Add Subscriber” button.
On the “Configure subscriber” panel, provide values for the name, description , host name and port. Select the protocol based on the analytics platform, whether you will use CDP or IBM z Operations Analytics applications, and whether or not you will secure communications with the subscriber using SSL. Click on “OK” and “UPDATE SUBSCRIPTIONS”.
Click on the “subscribe” icon of the subscriber. Select all the streams you want to be sent to this subscriber. Click on the “UPDATE SUBSCRIPTIONS” button.
Save the policy.
Migrating from previous PTF
The Data Streamer, Log Forwarder and System Data Engine (SDE) will support policies created in the Configuration Tool on previous continuous delivery PTFs with no changes to the policy. This means that the user can migrate the Data Streamer, Log Forwarder and SDE independently from the policy.
To migrate a policy from a prior PTF level of the Configuration Tool, simply open the policy in the 2Q 2019 level of the Configuration Tool and save the policy. The user will see some changes to the policy:
The UTF-8 transcription transform and any splitter transforms will no longer be displayed on the UI.
Tags on the stream icon are no longer displayed.
The protocol of the subscriber has been changed according to the following rules:
If the subscriber subscribes to streams under the IZOA heading, the new protocol will include IZOA.
If the subscriber subscribes only to streams under the Common Data Provider heading, the protocol will include CDP
If the data was sent to Logstash as “unsplit”, the new protocol will be “IZOA on IOA-LA via Logstash”
If the data was sent to Logstash as “split”, the new protocol will include “Elasticsearch via Logstash”
If the data was sent to the Data Receiver, the new protocol will include “via Data Receiver”
Generic HTTP, Generic HTTPS and CDP Debug are migrated without change
The secure status of the data will migrate unchanged
More information
See the interactive demo at https://common-data-provider.github.io/create_policy/.
For more information, see the official documentation at https://www.ibm.com/support/knowledgecenter/SSGE3R_1.1.0/welcome.html.