Written by Todd Havekost on November 29, 2023.

One of the key facets of the multitude of mainframe modernization journeys that are underway across the industry is providing easy access from off-platform applications to the extensive business-critical data and applications that reside on the mainframe. In today's world, that access is typically facilitated through APIs (Application Programming Interfaces). APIs provide a simple and intuitive way for developers to access mainframe transactions and data using interfaces that are familiar to them, without needing to learn the specifics of z/OS or its subsystems. This approach is also attractive because it enables reuse of existing mainframe functionality without the need to modify applications or replicate data. 

There is also business value in having this access going both directions. In addition to distributed or cloud-native applications accessing mainframe data and capabilities, the functionality of mainframe applications can also be enhanced through the ability to call off-platform applications. 

Over the years, subsystem-specific solutions have been developed and utilized to address this need. Examples of these solutions include CICS Transaction Gateway (CTG) and the IMS Mobile Feature Pack. But the value of having a single solution that interfaces across the breadth of the platform is apparent. 

z/OS Connect EE has emerged as IBM's widely-adopted strategic solution providing a single-entry point across Z platform subsystems. It handles converting incoming messages to the distinct formats expected by each subsystem, as well as centralizing administrative tasks such as configuration and security. 

z/OS Connect EE exposes both z/OS resources to the cloud via RESTful APIs (in which case z/OS Connect EE is considered the API “Provider”), and external REST APIs? in the cloud to z/OS applications (for which zZOS Connect EE functions as the API “Requester”). ‘You might consider zZOS Connect EE to be conceptually a little like the z/OS Communications Server; an IBM-provided system component facilitating connections of API services to and from the mainframe, handling all the connection logistics including data conversions. 

Measuring z/OS Connect EE Activity 

As you would expect from major products executing on the mainframe, z/OS Connect EE generates measurement data in the form of SMF 123 records. Metrics provided by these records include request rates, the “System of Record” (SOR, e.g., CICS, Db2, IMS, MQ) participating in the request, the request method (e.g., GET, POST), HTTP response code, and response times by component within z/OS Connect EE and the SOR (see detail below). These metrics are commonly viewed by API and service names. Interestingly, the ‘Version 2’ records contain information for up to 20 requests per record®. However, one thing that is noticeably absent from this SMF data is any information about CPU consumption. 

Note: Visibility into CPU usage by z/OS Connect EE would be via SMF 30 address space data for the z/OS Connect EE servers (and could also be captured in RMF 72 report classes). There is no request-level reporting of CPU consumption, however CPU usage by individual requests is likely to be minimal. Also, z/OS Connect EE is Java-based, so a high percentage of product CPU is eligible to be offloaded to a zIIP, meaning that if it did consume large amounts of CPU time, that would probably not be as concerning as if it was using general purpose CPU time.

Total CPU consumption by z/OS Connect EE could come into play if the volume of API requests was very large. In that case, one course of action could be to analyze payload data and see if the data requests could be reduced - the zZOS Connect EE SMF records do provide information about the request payload. Address space level data could also help sites to compare the overall CPU cost of using z/OS Connect EE versus the cost of moving the data off platform. 

Since there are numerous differences in function between the Provider and Requester use cases, their measurement data is captured in separate SMF 123 record subtypes: 

• SMF 123.1 - Provider
• SMF 123.2 - Requester

The SMF record layouts can be accessed at these IBM URLSs (or in the z/OS Connect Enterprise Edition 3.0 manual): 

• Provider data: SMF 123.1 V2 record layout.
• Requester data: SMF 123.2 V2 record layout. 

Note that even though the “version 2” of both subtypes supersedes “version 1” and contains more detailed information, the default is still to produce version 1 records, presumably for compatibility reasons. Sites are encouraged to ensure they are generating the higher-value version 2 records by configuring the z/OS Connect EE audit interceptor to setthe apiProvidersmfversion and apiRequesterSmfVersion attributes to 2" for the API provider and API requester record types, respectively. 

Make sure to enable the Version 2 format of the z/0S Connect EE SMF records. 

Measuring z/OS Connect EE Provider Activity 

The SMF 123.1 data enables sites to view z/OS Connect EE Provider activity from many perspectives. Figure 1 on page 5 shows the overall workload flow, the timing measurement points (indicated by the “Tn’ numbers in the diagram), and highlights of the metrics available for Provider activity (in green) and for both Provider and Requester activity types (in blue). 

Figure 1 - API Provider Metrics (123.1) 

Common starting points for views of Provider activity might be request rates by system ID or by APl name. Figure 2 moves to the next level of detail and indicates request counts per reporting interval by Service Name for a selected (higher volume) API. In this example, the request is only using a single service, but in other application designs an APl may reference several distinct services. 

Figure 2 - Number of Requests for Selected APl Name by Service Name (© IntelliMagic Vision) 

This chart is “By Service Name”. In this example, the API only calls a single service. If it called three distinct services, they would all appear here.

Recommendation: It is my understanding that current API design best practices recommend a 1-to-1 relationship between API name and Service name.

Figure 3 shows activity for a selected API by Service Provider Name and System of Record (SOR) Identifier. 

Figure 3 - Number of Requests for Selected APl Name by Service Provider Name and SOR Identifier (© IntelliMagic Vision) 

In this scenario, CICS is servicing the request, and the CICS applid is included in the SOR identifier. Notice that almost all the requests here are being serviced by applid AO1CIC70 (the red line) and only during sporadic intervals by the other two applids (as indicated by the yellow and blue dots). 

Sites may also be interested in viewing the frequency of requests that encounter non-standard HTTP response codes (values other than 200). In the example in Figure 4 on page 7, only a couple of non-standard 404 responses occur (represented by the blue dots during the 0230 and 2015 intervals). 

Figure 4 - Number of Requests for Selected APl Name by HTTP Response Code (© IntelliMagic Vision) 

Request and response payload lengths are also captured in both the Provider and Requester data. Sample reporting of payload length data is shown later, in ‘Measuring z/OS Connect EE Requester Component Response Times' on page 13. 

Measuring z/0OS Connect EE Provider Component Response Times

The request flow and response time components when z/OS Connect EE serves as the Provider were shown in Figure 1 on page 5. The SMF descriptions (and field names) of the four captured timestamps are listed here: 

• SMF123S1_TIME_ZC_ENTRY - Time the request was received by the z/OS Connect EE server.
• SMF123S1_TIME_SOR_SENT - Time the request was sent to the system of record.
• SMF123S1_TIME_SOR_RECYV - Time the response was received from the system of record.
• SMF123S1_TIME_ZC_EXIT - Time the response was ready to be sent to the HTTP client. 

These four timestamps enable calculation of three timing components, as displayed in Figure 5 on page 8 (along with the number of requests in green). 

Figure 5 - Response Time Components for Selected APl Name (© IntelliMagic Vision) 

During this interval there were 4 larger spikes in the “Time Spent in SOR” component (in blue). Viewing that timing component by System of Record Identifier (CICS applid) (see Figure 6 on page 9) shows that those spikes all correspond to times when a request was served by a region other than the AO1CIC70 applid (in red) that handles almost all the requests (as we saw earlier in Figure 3 on page 6). Blue and yellow dots (indicating isolated intervals) occur at the four times corresponding to the larger spikes in overall response times. So if those longer response times are causing business impact, one course of action might be to ensure all requests are served by AO1CIC70. 

Figure 6 - Time Spent in SOR for Selected APl Name by System of Record Identifier (© IntelliMagic Vision) 

Measuring z/OS Connect EE Requester Activity 

Activity when z/OS Connect EE is facilitating calls from z/OS applications to remote services (likely in the cloud) are captured in the SMF 123.2 Requester data. Figure 7 shows the (1) overall workload flow, (2) timing measurement points (indicated by the “Tn” numbers in the diagram), and (3) highlights of the metrics available for Requester activity (in purple) and for both Requester and Provider activity types (in blue). 

Figure 7 - API Requester Metrics (123.2) 

Many of the available activity views are similar to those we have already seen from the Provider data. In this section we will cover metrics unique to Requester activity, which include: 

• Job Name of Calling Application (e.g., BATJOBO1).
• API Endpoint Reference (e.g., GATEWAYSCOREAPI).
• Host of API Endpoint (e.g., https://scoreapi.acct.fin).

Another potential Requester view of interest is activity by Method for Endpoint Request. Figure 8 shows that POST and PUT methods are issued for the selected API Requester. 

Figure 8 - Number of Requests for Selected APl Requester Name by Method for Endpoint Request (© IntelliMagic Vision) 

When the requests are being issued from CICS, captured data includes the APPLID of the CICS region, the job name (in this case the region name), and the transaction ID. 

Figure 9 - Number of Requests for Selected API Requester Name by CICS APPLID and Job Name and CICS Tran ID (© IntelliMagic Vision) 

In Figure 9, almost all the activity comes from transactions PM02 and PM03 in APPLID A01CIC70 with address space name P03700D (red and blue lines), with occasional calls from APPLID A01CIC72 (yellow and green dots). For requests issued by IMS, available data includes IMS region ID, application name, Program Status Block (PSB) name, and transaction ID. Requester data also provides visibility into the Path Invoked on the API Endpoint, as shown in Figure 10 on page 12. 

Figure 10 - Number of Requests for Selected API Requester Name by Path Invoked on API Endpoint (© IntelliMagic Vision) 

The length of payloads sent to and received from the API endpoint is captured in the Requester data and may serve to establish profiles. The dashboard excerpt shown in Figure 11 reveals consistent profiles by CICS transaction ID and method. In this example, the number of bytes sent is very consistent: 830 for PM02 POST, 42 for PM03 POST, 24 for PMO3 PUT. Bytes received back from the endpoint are also relatively consistent: 160 for PMO02 POST, and near 60 for both methods for PM03. 

Figure 11 - Payloads Sent & Received for APl Requester Name by CICS Tran ID & Method for Endpoint Request (© IntelliMagic Vision) 

Measuring z/0S Connect EE Requester Component Response Times 

The request flow and response time components when z/OS Connect EE facilitates calls to a remote API from a z/OS application were illustrated earlier in Figure 7 on page 9. In this scenario the zZOS Connect EE SMF 123.2 records capture five timestamps, as detailed here. 

• SMF123S2_TIME_STUB_SENT - Time when the request left the calling application.
• SMF123S2_TIME_ZC_ENTRY - Time when the Z/OS Connect EE server received the request.
• SMF123S2_TIME_ENDPOINT_SENT - Time when z/OS Connect EE sent the request to the endpoint.
• SMF123S2_TIME_ENDPOINT_RECEIVED - Time when z/OS Connect EE received a response from the endpoint.
• SMF123S2_TIME_ZC_EXIT - Time when the response was sent to the requester. 

Note that five timestamps are captured in this workflow, one more than in the Provider scenario, enabling calculation of four timing components, shown in Figure 12 (along with the number of requests in light blue). In Requester data we have seen to date, most of the elapsed time is spent in the endpoint (in yellow), as is the case here. 

Figure 12 - Response Time Components for Selected APl Requester Name (© IntelliMagic Vision) 

References 

z/OS Connect EE is relatively new (in mainframe terms), so you might not be very familiar with it. Hopefully the following documents and websites will be some help for both new and experienced readers: 

• 2017 TechChannel article titled ‘Connect z/0S Applications and Data to the Hybrid Cloud With APIs by Bruce Armstrong. This article is a little old, but it provides an excellent introduction to how z/OS Connect plays a role in modernizing your mainframe environment - great for people that are not yet familiar with zZOS Connect.
• IBM White Paper, /BM z/0OS Connect Performance Evaluation’, by Kevin Hite and Richard Antikoll.
• IBM Wildfire workshop, “ZCINTRO - IBM z/OS Connect: An introduction and overview”, by Mitch Johnson and John Brefach.
• IBM TechU 2021 presentation, “z/OS Connect EE - Performance Considerations to Highlight the Hybrid Cloud World®”, by Shalawn King and John Burg.
• IBM manual zZOS Connect Enterprise Edition 3.0. There appears to be only one manual that covers all aspects of zZOS Connect EE, but it is 946 pages long.
• IBM Content Solution webpage for z/OS Connect Enterprise Edition. 

Summary 

We want to thank Todd and Dennis for their great work on this article. From our perspective, z/0S Connect EE seems to be a little like MQ - from a relatively quiet start, it grew to the point that it is now in use in nearly every customer we work with. Given that it plays a pivotal role in advancing the use of APIs to make your z/OS applications and data available to the outside world, it is vital that it is understood and that it delivers sparkling performance. Dennis and Todd’s article will hopefully help you in achieving that objective. 

While many customers are now using z/OS Connect EE, there still seem to be challenges with limited visibility into the SMF 123 data it generates. This article shows some real world examples of how the z/OS Connect EE SMF data can be used. And Todd and Dennis look forward to working with zZOS Connect EE users going forward to gain further insights into ways SMF 123 data can help you better manage your z/OS Connect EE implementations and leverage that data to enhance availability and performance.