Written by Todd Havekost on February 19, 2020.
IBM continues to make significant hardware and software investments across the Z platform to enhance data security, highlighted with the pervasive encryption initiative that was introduced with the z14 processor. Implementing encryption of all data across its entire lifecycle is a comprehensive undertaking, and not all sites will have the business requirement for that level of end-to-end encryption. However, encrypting network traffic between the mainframe and the outside world so that it cannot be intercepted is a requirement for almost every company.
With the intense focus on securing customer data in today's enterprises, one of the most challenging jobs has to be that of the network administrator, who is responsible for ensuring that all data being transmitted (especially outside the data center) is encrypted as prescribed by corporate security standards. We mainframers, being accustomed to leveraging SMF metrics that give us insights that far exceed other platforms, may not have been aware of the huge visibility gap for network encryption that existed with the tools previously available. I imagine that task would have been like trying to plug a never-ending series of plumbing leaks while working without almost any light.
But beginning with z/OS 2.3, IBM provides the capability to have an entirely new level of visibility into the encryption status of all network traffic to and from your z/OS systems. This is enabled through z/OS Encryption Readiness Technology (zERT), which makes this data available in SMF 119 records. Because managing network encryption is such a critical need, and the potential visibility provided by zERT summary data is so well-designed and comprehensive, I consider the introduction of zERT SMF 119.12 records to be perhaps the greatest advance in SMF data since CPU Measurement Facility (CPUMF) began providing instrumentation of processor cache metrics through SMF 113 records in 2008.
The goal of this article is to introduce you and your colleagues to this relatively new and very valuable data source, and to suggest some approaches to increase the benefit you can derive from this data.
Technical Background
Figure 1 shows the four mechanisms provided by z/OS to cryptographically protect TCP/IP traffic.
Figure 1 - TCP/IP cryptographic protection on z/OS (© IBM Corporation)
These mechanisms as numbered on the diagram are:
- TLS/SSL direct usage
- Application Transparent TLS (AT-TLS)
- Virtual Private Networks using IPSec and IKE
- Secure Shell using z/OS OpenSSH
What makes zERT so powerful is that it leverages the fact that the TCP/IP stack serves as a central collection point and repository for all cryptographic protection attributes. Think of it as giving parents the capability of being plugged in to their child's entire smart-phone social media feeds 24x7, and thus always being aware of any potentially harmful influences in their lives.
As you can imagine, the information provided by zERT can be very helpful to network administrators responsible for managing the encryption levels and protocols in place for network data sent to various IP addresses inside and outside the enterprise. Perhaps most importantly, it identifies what network traffic is not being protected (either at all, or with a recognized protocol). It also captures how the traffic is being protected, and, if follow up is needed, who the traffic belongs to. Thus, zERT data can enable network administrators to both evaluate ongoing adherence to security policies, and programmatically provide data for required reporting to auditors and compliance officers.
zERT Data Logistics
The initial zERT capability was delivered with z/OS 2.3, and generated “Detail” data - one SMF 119.11 record for every session. But this resulted in extremely high data volumes, so new function APAR PI83362 delivered the capability to generate zERT “Summary” data: one type 119.12 record per SMF interval for each unique session type, between each unique pair of server and client IP addresses. These Summary records are well-designed and more than sufficient for typical analysis. They are controlled by TCP/IP Profile statements:
- GLOBALCONFIG ZERT AGGREGATION, which activates zERT in-memory monitoring, and
- SMFCONFIG ZERTSUM, which causes zERT SMF 119.12 summary records to be written.
Detail records (119.11) are suppressed by default, and only generated if you specify SMFCONFIG ZERTDETAIL; the default is NOZERTDETAIL. As far as we can tell, nearly no one enables the detail records because of the massive volume of records that are produced.
When zERT is activated, the TCP/IP stack will create a zERT entry for each unique session type between client/server pairs during each SMF interval. It may contain detailed information about the session protocol - this information is provided via an interface with the cryptographical protocol provider (such as System SSL, OpenSSH or IPsec). Or it may obtain information by observing the stream for TLS, SSH, or SSL sessions, in which case fewer details tend to be available.
Based on this information, the zERT software will classify the protection of a session as either None (which means no protection was recognized), TLS, SSH, or IPsec. zERT also identifies a few special cases, such as Enterprise Extender sessions, as well as output [IPv4] sessions from an FTP server.
The zERT Summary data contains a wealth of information. Along with the protocol, zERT records contain other protection attributes including protocol version, the cryptographic algorithm being used, and key lengths. zERT also collects identifying attributes to track connections between each pair of Client and Server IP addresses, including port number, job name, and user ID.
Figure 2 shows an example of the data provided from a zERT Summary record for data encrypted with the TLS protocol (with data from a single record broken into 2 lines for readability), as presented by IntelliMagic Vision. IBM and other vendors also provide capabilities to process zERT data, through SMF 119 records or network management interfaces. IBM provides the zERT Network Analyzer plugin for z/OSMF with new function APAR PH03137 to analyze zERT Summary data. This is a new plugin that provides a web UI to make zERT data consumable. It enables z/OS network administrators to create queries to analyze specific systems, endpoints, time spans, and security attributes of interest. [Editor’s Note: The zERT z/OSMF plugin uses a Db2 database, so it requires additional Db2 setup before it can be used - this is a little more complex than most z/OSMF plugins.]
Figure 2 - TLS Specific Protocol Information (© IntelliMagic Vision)
Analysis of zERT data also involves translating the more than 600 binary and EBCDIC codes that appear in the SMF 119.12 records into readable text. Coded fields that have been translated in Figure 2 include cipher suite, encryption algorithms, and message authentication types.
The zERT Summary records also contain connection and throughput counters, as seen in Figure 3 on page 6. These include the total number of connections, the number of partially protected connections (where encryption was not applied during the entire session), and the number of short (less than 10 seconds) connections. Short connections are especially interesting for TLS, where establishing the session is expensive in terms of CPU consumption.
Figure 3 - zERT Connection and Throughput Counters (© IntelliMagic Vision)
It is very important to note that zERT does not collect or record the values of keys, initialization vectors, or any other secret values that are exchanged or negotiated during the session.
Interpreting and Gaining Value from zERT Data
As with most SMF data sources, the zERT records are a rich source of great data, but a significant analysis effort can be required to derive value from that data. One important consideration is that different classes of network traffic are likely to have differing security requirements.
Classifying Network Traffic
When you start investigating zERT data, you may quickly learn that most of the captured traffic occurs inside your organization, not even leaving your mainframes in many cases. Much of the traffic within z/OS sysplexes occurs because TCP/IP has come to be the protocol of choice for communication between many applications (taking the place of more native z/OS protocols like XCF). In some cases, this traffic may never even hit a network interface, as is the case when it is transmitted via HiperSockets or SMC-D (Shared Memory Communications - Direct Memory Access), which leverage processor memory to handle TCP/IP traffic between LPARs residing on the same processor. For network traffic that never leaves the mainframe, implementing encryption may not be your top priority [Editor's Note: I asked Todd if the zERT SMF records contain any information about the CPU cost of encrypting or decrypting the traffic going up and down to each IP address. Unfortunately (for us performance people), the records do not contain any of that type of information - they are very much focused on what type of encryption (if any) is being used.]
You are also likely to find a high volume of traffic for applications that interact between the mainframe and other platforms within your data center, such as Linux or Windows servers. This traffic will never cross the corporate firewalls, and hence may also be subject to lower security requirements than traffic to the public Internet.
Traffic that does travel outside your corporate Intranet can be treated as one big pool, but your organization may have some strategic partners that handle critical, high-volume traffic, with whom you may have special arrangements. One “partner” example could be connections from retailers or banks to credit card processors.
Finally, you are likely to be managing a large pool of public IP addresses. Having easy visibility to differentiate among all these different classes of network traffic may be very helpful as you administer their potentially varying security requirements. The approach taken by IntelliMagic Vision to help facilitate this analysis is to let you specify control statements that categorize your network traffic based on IP address ranges. Traffic can be categorized into these groupings:
- Sysplex - network traffic between mainframes (within a sysplex or across sysplexes)
- Local - traffic to other platforms within the data center
- Partner - external traffic with special partners (as described above)
- Public - all other external traffic
Figure 4 illustrates throughput by traffic class (Intra-Sysplex, Local, Public) for each protocol type (as they appear in the legend on the right). In this example, most of the data is Local (between mainframe and other platforms within the data center). And most of that Local throughput is either not encrypted (blue bar) or encrypted with TLS v1.2 (yellow bar).
Figure 4 - Throughput by Traffic Class (© IntelliMagic Vision)
Dynamic Navigation
A likely next analytical step arising from the information shown in Figure 4 on page 7 is to identify the network traffic that is currently not protected and who that traffic belongs to. This is a prime example of the value of using an interface that allows you to dynamically navigate through the data with context-sensitive drill-downs, enabling you to decide based on the current display the next path you wish to pursue.
Figure 5 displays one set of IP server/client address pairs derived from drilling down on traffic with encryption protocol “NONE”. It also includes an example of a dialog box (in the middle of the screen) that provides many options for further investigation of any of the server IP (previously selected via product navigation and reflected in the report heading) and client IP (listed across the X-axis at the bottom) pairs currently displayed. The next analytical step could focus on “Sockets” (address space names), User ID, or many other attributes for the selected server/client address pair.
Figure 5 - Example Report Showing Drill-down Options (© IntelliMagic Vision)
References
You can find more information about zERT and the role it plays in managing encryption of mainframe network traffic in these reference documents, presentations, and videos:
- IBM Manual z/OS Communications Server IP Programmer's Guide and Reference, SC27-3659 (Appendix E documents the record layout of the SMF 119.12 record)
- YouTube video z/OS Encryption Readiness Technology (zERT) by IBM's Chris Meyer.
- SHARE 2020 in Ft. Worth Session 26138, Pervasive Encryption: Get a Grip on Your z/OS Network Encryption with zERT, by Chris Meyer and Al Shakra.
IntelliMagic will be briefly demonstrating zERT visibility through IntelliMagic Vision in two sessions at the upcoming February 2020 SHARE in Ft. Worth:
- Session 27094, Why Now is the Time to Replace Legacy RMF/SMF Reporting Tools and Processes, and
- Session 27128, 2020 Vision Into z/OS Infrastructure Operation and Availability, both sessions by Brent Phillips and Todd Havekost
You can also find a video demonstrating using IntelliMagic Vision to analyze zERT data, and information about a new z/OS Network Encryption Assessment Services offering here.
Summary
IBM's introduction of z/OS Encryption Readiness Technology is a game-changing advance for network administrators responsible for managing the deployment of encryption in their environments. I am aware of network team members at one site who have blocked weekly time on their calendars to analyze zERT data and drill-down into the details of any unencrypted or inadequately encrypted traffic in their environment, generating action items to resolve any identified gaps. I hope your network administrators are currently (or soon will be) leveraging the visibility available through the zERT data to protect your company from exposure arising from unencrypted network traffic.
[Editor’s Note: We hope that you enjoy Todd’s article and share it with your network administration colleagues. I’m sure they will appreciate the insight that zERT gives them, and the time that this will save them. And who knows, you might even win over a few more SMF fans! We want to give a big Thanks to Todd for providing us with this valuable addition to our Tuning Letter.]