Abstract
Since its inception in 2016 IBM Z OMEGAMON for JVM has been helping customers identify performance issues in Java applications running on z/OS. With a focus on z/OS Connect EE, V5.5.0 FP2 expands on the detailed monitoring of APIs, now including API requester statistics.
In January 2021, Fix Pack 2 (FP2) for IBM Z OMEGAMON for JVM, V5.5.0 was released in PTF UJ04755. This fix pack introduces the highly anticipated support for z/OS Connect Enterprise Edition (EE) API requesters. Also known as “outbound” APIs this z/OS Connect EE feature allows z/OS native applications written in languages such as COBOL and PL/1 for CICS, IMS or batch to be able to call RESTful APIs. It accomplishes this using a simple, callable stub module (BAQCSTUB) without having to implement all the REST API services, JSON parsers and so forth. z/OS Connect EE transforms the z/OS application request parameters into JSON and makes the API request on behalf of the application. The endpoint API returns a response in JSON which z/OS Connect EE then transforms into a native format to be returned to the application.
Figure 1: z/OS Connect EE used as an API Requester
New code in IBM Z OMEGAMON for JVM now allows customers to monitor requester API calls as they are being processed by z/OS Connect EE. OMEGAMON records the response times for each leg of the request including the time taken to authorize requests; the time spent in z/OS Connect; and how much time is spent in the API endpoint. It presents this data in the Tivoli Enterprise Portal (TEP) or enhanced 3270 user interfaces (e3270UI) as soon as a request is completed. Summary reports are provided in addition to individual metrics for every request made. Any aberrant response times can be highlighted and acted on through the ITM Situation facility, allowing the right staff to be alerted and resolve the issue using the information provided by OMEGAMON. Figure 2 shows an example report of API requester calls made to a monitored z/OS Connect EE server in the previous five minutes.
Figure 2: API Requester Summary Workspace in e3270UI
This workspace shows the total number of requests made during the period; the number of those requests that ended in error; the average response time of all requests, and the minimum, maximum and standard deviation of the response time. Response time statistics are further divided into average time spent in z/OS Connect EE and the time spent in the endpoint application.
What’s more, the product also provides pre-z/OS Connect time. This is the amount of time between the native application call to the z/OS Connect EE stub and the request arriving in z/OS Connect. If you want to know where your API request calls are spending all their time, look no further. There are also reports summarized by origin (type and jobname), and endpoint API so you can compare different endpoints and isolate the response time hogs.
Zoom into any of the summary rows and you get a list of every individual requester call made during the period. Sort the results by any column or use one of the supplied filters to reveal individual outliers. Select any record for the minute details that will point you to the root of a problem.
Figure 3: Requester Detail Workspace in e3270UI
If the server uses open authorization V2 (OAuth2) tokens or Java Web Tokens for user authorization , the product also reports the time taken to acquire tokens or create new, including any token server retry time if a token has expired. This is shown in Figure 3 above under Request Access Token Time.
Inflight APIs
If you want to know what’s running right now in your z/OS Connect EE server, then look no further. New in fix pack 2 is an inflight view of all provider APIs currently being executed.
Figure 4: TEP Workspace Showing Inflight API Provider Calls
Figure 3 shows a product-provided TEP workspace that renders inflight requests executing on the selected. It includes the total amount of time the request has been running so far, and which phase of processing the provider API is in. If the request has just been received by z/OS Connect EE but not yet transformed and sent to the system of record application, the phase is shown as “PRESOR”. If the service provider for the API has invoked the system of record application, then the phase is “SOR”. Similarly, if a request is caught just after it has returned from the system of record (but before a JSON response has been sent to the API caller), the phase is shown as “POSTSOR”. Once the request hits the system of record, the accumulated time spent in the SOR is also shown. Note: pre-and post-SOR times are calculated at the ITM agent and so may differ slightly from the final time posted to the log stream data store.
Early Failures
Another enhancement in FP2 is the capture of so-called “early failures”, although it is likely to go unnoticed. In previous versions of OMEGAMON for JVM (and prior to z/OS Connect EE V3.0.39), requests rejected due to malformed requests or because of authorization failures would not be recorded by the product. This is because these types of errors are not processed by z/OS Connect but by the Liberty hosting layer. With FP2, requests rejected by early failure are included in the summary and detail reports and included in the error counts, with HTTP status codes greater than 200.
Figure 5: Example of Early Failure (Unknown Resource)
IBM OMEGAMON for JVM is part of IBM’s Z monitoring and management suite ( IZMS ) that delivers the capabilities to detect and identify potential issues before they disrupt your business. Detect is one of the three areas of best practices and capabilities that you need to integrate to help accelerate your journey to AIOps that includes IBM Z.
Download
IBM Z OMEGAMON for JVM, V5.5.0 Fix Pack 2 PTF UJ04755 is available for download from the IBM Shopz web site at https://www.ibm.com/software/shopzseries/ShopzSeries_public.wss
Application support for Fix Pack can found on the IBM Support site at:
https://www.ibm.com/support/fixcentral/swg/selectFixes?product=ibm%2FTivoli%2FIBM+OMEGAMON+for+JVM+on+zOS&fixids=5.5.0-TIV-KJJ-FP0002&source=dbluesearch&function=fixId&parent=ibm/Tivoli
About the author.
Nigel Williams is a Senior Software Engineer with Rocket Software, Inc. He is lead developer and architect of IBM Z OMEGAMON for JVM.