MQ

 View Only

MQ for z/OS using AT-TLS to provide channel encryption

By Anthony Sharkey posted Tue October 12, 2021 02:14 AM

  

Many years back, I ran some performance comparisons for securing and encrypting data over MQ channels - comparing MQ with System SSL/TLS ciphers, against regular MQ channels protect using  Application Transparent Transport Layer Security (AT-TLS), with the results documented in the performance report "MP16 - Capacity Planning and Tuning Guide".

In MQ for z/OS v9.2, we introduced support for TLS 1.3 ciphers and the performance is discussed in the MQ for z/OS v9.2 performance report.

Given the age of the AT-TLS performance measurements and added support for TLS 1.3, we thought it was about time that we updated the AT-TLS measurements using the  latest ciphers available.

The performance evaluation will be updated in MP16 in slightly shorter form that appears below:

Using AT-TLS to encrypt data flowing over IBM MQ channels

 

Application Transparent Transport Layer Security (AT-TLS) is based on z/OS System SSL, and transparently implements the TLS protocol (defined in RFC 2246) in the TCP layer of the stack.

 

When running channels between queue managers hosted on z/OS, AT-TLS can be used to encrypt messages transported over MQ channels rather than relying on IBM MQ channels performing the encryption function. The use of AT-TLS can result in reduced costs within MQ.

 

It should be noted that whilst AT-TLS can offer reduced costs, these are not always reflected in improvements in the rate that message data is transferred, particularly when using encryption ciphers that are unable to exploit hardware acceleration, such as TLS 1.3 cipher TLS_CHACHA20_POLY1305_SHA256. IBM MQ for z/OS performance will also be affected when using encryption ciphers that are unable to exploit hardware acceleration, but the impact may be reduced due to MQ using separate tasks for encryption and networking.

 

Who pays for AT-TLS?

MQ channels with SSLCIPH configured will see the encryption/decryption cost associated with the channel initiator address space.

 

When transporting messages using channels encrypted using AT-TLS, the cost of encryption is charged to the callers unit of work (i.e. the channel initiator) and decryption is incurred by the TCPIP address space as the decryption is performed by an SRB running in the TCPIP address space. 

Limitations

IBM MQ allows the user to specify different SSL cipher specifications for each channel.

To run with different cipher specifications using AT-TLS can involve defining additional rules plus either specifying the LOCLADDR attribute on the channel to restrict the port being used or by running with multiple listeners defined on the channel initiator.

 

IBM MQ allows the secret key negotiation to be performed when the number of bytes sent and received within an SSL conversation exceeds the SSLRKEYC value, whereas AT-TLS allows the renegotiation to take place after a period of time has been exceeded.

 

When the AT-TLS encryption is performed, the TCP socket is blocked – this can have a noticeable effect on throughput rate with large messages even when dynamic right sizing is enabled on the TCPIP stack.

 

Channels protected with CHLAUTH rules may not be allowed to start if the rule contains a value for SSLPEER.

 

Performance comparison

The following measurements were run using a 1Gb network between 2 z/OS v2r4 LPARs each with 3 dedicated processors on a IBM z15 (8561-7J1).

 

A request/reply workload between 2 queue managers was run over a pair of sender-receiver channels using non-persistent messages.

 

In the measurements using channels with SSLCIPH cipher specifications, the SSL key negotiation has been disabled (by setting to ’0’) in order to provide a direct comparison. Similarly the AT-TLS negotiation period has been disabled.

 

The costs shown in the following charts are for the queue manager, channel initiator and TCPIP address spaces only and are based on the cost observed in both LPARs.

 

For the purposes of these measurements we are using the following ciphers:

Type

MQ SSL/TLS cipher

AT-TLS cipher

TLS 1.2 (hardware - CPACF)

TLS_RSA_WITH_AES_256_CBC_SHA256

TLS_RSA_WITH_AES_256_CBC_SHA

TLS 1.3 (hardware - CPACF)

TLS_AES_256_GCM_SHA384

TLS_AES_256_GCM_SHA384

TLS 1.3 (software)

TLS_CHACHA20_POLY1305_SHA256

TLS_CHACHA20_POLY1305_SHA256

As discussed in the MQ for z/OS v9.2 performance report, the cost of encrypting and decrypting data flowing over MQ channels is typically reduced with the use of CPACF (Central Processor Assist for Cryptographic Functions) and is relatively consistent, however cipher TLS_CHACHA20_POLY1305_SHA256 is not supported by hardware encryption and must be processed using software, hence for the purposes of this report we are calling the cipher “TLS 1.3 (software)”.

2KB Non-persistent request/reply workload

2KB Request/Reply workload

Notes on 2KB workload:

  • Cost of transporting the message is up to 25% lower when using AT-TLS to protect the message, depending on cipher used.
  • Channel initiator costs are up to 45% lower when using AT-TLS.
  • TCP/IP costs are 1.5-1.6 times higher when using ciphers able to exploit hardware encryption with AT-TLS.
  • TCP/IP costs are 24 times higher when using ciphers that rely on software encryption with AT-TLS when compared to the equivalent TCP/IP costs on MQ channels protected using SSLCIPH.


16KB Non-persistent request/reply workload

16KB non-persistent request/reply

Notes on 16KB workload:

  • Chart uses a log scale – for the AT-TLS using TLS 1.3 (software) the costs attributed to the channel initiator and the TCP/IP address space are approximately equal at 2.2 CPU milliseconds per transaction.
  • Cost of transporting the message is up to 42% lower when using AT-TLS to protect the message, depending on cipher used.
  • Channel initiator costs are up to 55% lower when using AT-TLS.

32KB Non-persistent request/reply workload

32KB request/reply workload

Notes on 32KB workload:

  • Chart uses a log scale – for the AT-TLS using TLS 1.3 (software) configuration, the costs attributed to the channel initiator and the TCP/IP address space are approximately equal, at 4.3 CPU milliseconds per transaction.
  • Cost of transporting the message is up to 34% lower when using AT-TLS to protect the message, depending on cipher used.
  • Channel initiator costs are up to 50% lower when using AT-TLS.

64KB Non-persistent request/reply workload

64KB request/reply workload

Notes on 64KB workload:

  • Chart uses a log scale – for the AT-TLS using TLS 1.3 (software) configuration, the costs attributed to the channel initiator and the TCP/IP address space are approximately equal, at 8.4 CPU milliseconds per transaction.
  • Cost of transporting the message is up to 34% lower when using AT-TLS to protect the message, depending on cipher used.
  • Channel initiator costs are up to 50% lower when using AT-TLS.

1MB Non-persistent request/reply workload

1MB request/reply workload

Notes on 1MB workload:

  • Chart uses a log scale – for the AT-TLS using TLS 1.3 (software) configuration, the costs attributed to the channel initiator and the TCP/IP address space are approximately equal, at 133 CPU milliseconds per transaction.
  • Cost of transporting the message is up to 42% lower when using AT-TLS to protect the message, depending on cipher used.
  • Channel initiator costs are up to 60% lower when using AT-TLS.

Is the reduced cost reflected in a throughput improvement?

As the previous measurements have shown, there is the potential for reduced CPU cost when transporting MQ data over networks protected by AT-TLS encryption rather than protecting MQ channels using the SSLCIPH channel attribute.

 

Those costs differences vary depending on the message size and the cipher used. In our measurements the cost differences were:

  • TLS 1.2 (hardware) – between 18-25% lower cost using AT-TLS
  • TLS 1.3 (hardware) – between 25-43% lower cost using AT-TLS
  • TLS 1.3 (software) – between 1-2% lower cost using AT-TLS

 

When hardware can be exploited, we are seeing a cost saving from using AT-TLS rather than MQ and System SSL services - but this saving is negligible when using ciphers that have to be processed in software cycles.

 

In terms of transaction rates, the benefits are less clear - generally the round-trip times are similar when using ciphers able to exploit hardware encryption, and certainly using the ciphers able to use hardware support, we are hitting network limitations. It is worth noting that this is despite the reduced cost of the workload, suggesting the encryption/decryption of data using AT-TLS is hitting some inhibiting factor, whether waiting for resource or the time taken to encrypt/decrypt the data is causing the network window sizing to be less optimal.

 

Additionally, with regards to the TLS 1.3 cipher TLS_CHACHA20_POLY1305_SHA256, we are seeing the AT-TLS throughput degrade relative to the MQ with System SSL configuration, particularly as the message size grows and this is particularly notable as the measurements are neither CPU nor network constrained.

 

Table comparing average transaction round-trip times (microseconds) using TLS 1.3 (software) cipher TLS_CHACHA20_POLY1305_SHA256:

Message Size

2KB

16KB

32KB

64KB

1MB

MQ + System SSL

2172

11946

22699

44452

693844

MQ + AT-TLS

2199

12840

25342

51707

956592

% increase in round-trip time

+1.24

+7.48

+11.64

+16.32

+37.87


Comparing the MQ channel status data for the 1MB measurements:

 

MQ + System SSL

AT-TLS

XBATCHSZ

5

5

NETTIME

2-8 milliseconds

28-130 milliseconds

 

In this instance the AT-TLS configuration is achieving the same batch size, but the NETTIME reported (amount of time, displayed in microseconds, taken to send an end of batch request to the remote end of the channel and receive a response minus the time to process the end of batch request)  is significantly higher – this is primarily as in the AT-TLS configuration, the NETTIME includes the data encryption and decryption time.


Using NETSTAT

Reviewing NETSTAT information gathered from the comparison of the 1MB measurements showed some differences in the network behaviour.

The following charts are created from output gathered by gathering the results of polling using NETSTAT with the following command

TSO NETSTAT ALL (CLI <MQ Channel Initiator>

Congestion Window

The congestion window is the value that is used when congestion is detected in the network to limit the amount of data that is sent by the local stack. This value represents the maximum amount of data that is sent without waiting for an acknowledgment from the remote socket.

Congestion Window for 1MB workload using CHACHA20_POLY1305_SHA256

The MQ and System SSL configuration shows a steady and smooth increase to the congestion window size, indicating there are no congestion issues occurring.

The AT-TLS configuration shows the congestion window grow and then reset as congestion issues occur.

Smooth trip time

This is the average amount of time it has taken for a packet to be sent and an acknowledgement to be received for this connection.
Smooth trip time for 1MB workload using CHACHA20_POLY1305_SHA256

Once more, the MQ with System SSL data shows a smooth trip time and this is backed up by a low smooth trip variance.

The AT-TLS configuration shows a much higher smooth trip time and a much higher smooth trip variance.

Send data queued

This is the number of bytes of data on the send queue waiting for the remote side to acknowledge.
Send Data Queued for 1MB workload using CHACHA20_POLY1305_SHA256

The AT-TLS measurement again shows a much more varied behaviour to MQ with System SSL, and in this instance suggests that the time taken to decrypt the data using AT-TLS is causing the network data to be queued, and as a result causes the inefficiencies in the congestion window and smooth trip time.

Why is there no improvement to transfer rate despite the transport cost being reduced?

Network latency is clearly big factor in the transfer rate, but in our configuration the latency is low as the LPARs are linked with less than 50 metres of fibre.


The model used by Comms Server and AT-TLS to protect the data is a simple one – an SRB per socket processes the request – this processing involves both the encryption and sending of the data. As a result the send of the data over the network cannot begin until the encryption of the data is complete.

Similarly at the receiving-side, the SRB will receive the data and decrypt it, before becoming available to receive more data.

With the extended time that the encryption and decryption takes when using AT-TLS, particularly with cipher TLS_CHACHA20_POLY1305_SHA256 with larger payloads, the SRB is blocked for longer periods.

 

By contrast, MQ has dedicated tasks for the network interaction (dispatchers) and separate tasks for the encryption workload (SSL tasks). This enables MQ to be able to run encryption of one message on an SSL task in parallel with sending another message (or indeed chunk of a large message) over the network using a dispatcher task.

Starting and stopping MQ channels

The rate and cost at which MQ channels are able to start and stop is particularly of interest where the channels are starting and stopping regularly, perhaps due to a sudden increase or decrease in workload.

 

Report "MP16 - Capacity Planning and Tuning Guide". and the MQ for z/OS v9.2  performance report, both provide example costs and achieved rates when starting and stopping MQ channels, with and without MQ ciphers. The information presented in those documents has been refreshed as part of this evaluation as the cost of starting TLS 1.3 MQ channels has been reduced since the GA of MQ for z/OS v9.2 due to a number of changes:

  • A substantial performance gain in ICSF FMID HCR77D1 which reduces the cost of the key share handshake.
  • Reducing the number of client key shares used – this code has been available in MQ for z/OS Continuous Delivery releases V922 and onwards.

 Note: As the performance evalutation has been run using the V9.2 MQ code, the benefit from reducing the number of key shares is not demonstrated here, but certainly on IBM z15, FMID HCR77D1 had a significant impact to the cost of channel start of up to 80% of the cost observed at MQ v9.2 GA.

This blog provides a comparison of the starting and stopping 4000 outbound MQ channels and 4000 inbound MQ channels, that have been defined between two z/OS queue managers hosted on separate LPARs of a single z15. 

The distance between the channel end-points will affect the rate at which a channel can start as a high latency network will increase the time taken for the handshake process, involving multiple flows between channel initiators, to complete.

 

As before, the channels are configured as defined below:

Type

MQ SSL/TLS cipher

AT-TLS cipher

TLS 1.2 (hardware - CPACF)

TLS_RSA_WITH_AES_256_CBC_SHA256

TLS_RSA_WITH_AES_256_CBC_SHA

TLS 1.3 (hardware - CPACF)

TLS_AES_256_GCM_SHA384

TLS_AES_256_GCM_SHA384

TLS 1.3 (software)

TLS_CHACHA20_POLY1305_SHA256

TLS_CHACHA20_POLY1305_SHA256

 

The AT-TLS configuration used in these measurements specifies the server-side HandshakeRole serverWithClientAuth in the TTLSEnvironmentAction statement.

 

Additionally, the channels protected using AT-TLS ciphers are configured to use default options in the TTLSEnvironmentAdvancedParms statement for:

  • Renegotiation
  • RenegotiationIndicator
  • RenegotiationCertCheck

Enabling certificate checking, and full or abbreviated renegotiation may affect the costs observed.

Start channel rate

MQ channel start rate

Enabling secure channels does have an immediate effect on the rate at which MQ is able to start the channels:

  • Channels secured using MQ with System SSL are able to start at between 33 to 50% of the rate of unsecured channels.
  • Channels secured using AT-TLS are able to start at 75% to 80% of the rate of unsecured channels.

Start channel cost

MQ channel start cost

With secure channels, MQ using System SSL does see a small increase in TCPIP costs but the additional cost is primarily in the channel initiator address space.

  • For the TLS 1.2 cipher used, this results in the channel initiator cost doubling that of an unsecured channel.
  • For the TLS 1.3 cipher where hardware support is available, the channel initiator cost is three times that of the unsecured channel.
  • For the TLS 1.3 cipher without hardware support, the channel initiator cost of the channel start is 3.5 times that of the unsecured channel.

 

For channels secured using AT-TLS, the impact is primarily in the TCPIP address space, and in terms of CPU cycles is typically far less than those observed in the MQ channel initiator.

  • For the TLS 1.2 cipher, the TCPIP cost increases from 30 to 210 microseconds.
  • For the TLS 1.3 cipher with hardware support, the TCPIP cost increases from 30 to 530 microseconds.
  • For the TLS 1.3 cipher without hardware support, the TCPIP cost increases from 30 to 690 microseconds.

 

It is also worth noting that with the current configuration, MQ is able to offload some of the cryptographic processing to the CryptoExpress hardware (both co-processor and accelerator) at channel start, whereas the certificates we have configured with the AT-TLS configuration are not using 
cryptographic hardware.

Stop channel cost

Costs attributed to the MQ channel initiator for stopping channels are of the order:

  • 670 CPU microseconds for unprotected channels, or channels protected by AT-TLS policies
  • 950 CPU microseconds for channels protected by MQ and System SSL.

 

TCPIP costs when stopping MQ channels were of the order:

  • Not protected by AT-TLS policies: 45 CPU microseconds per channel.
  • Protected by AT-TLS policies: 100-150 CPU microseconds per channel.

Should I use AT-TLS to provide encryption of my MQ channels?

Using AT-TLS to provide encryption of MQ channels may offer a substantial reduction in CPU cost in the MQ channel initiator when compared with using MQ’s own implementation using System SSL, and this typically results in a net reduction when including the additional CPU cost in the TCPIP address space.

 

With regards to the processing model that TCPIP uses, i.e. a single task per socket which encrypts and sends data, or receives and decrypts the data, despite the cost reduction there is not always a commensurate improvement in throughput. There are instances when MQ is able to exploit parallel processing when sufficient SSL tasks and dispatchers are available such that MQ can achieve parity or indeed higher throughput over the channels.

 

Using AT-TLS does allow the user greater control over the security settings implemented but as a result, does require further understanding of the ramifications of choosing particular options or indeed of relying on the defaults.

However, since AT-TLS is not fully integrated with MQ, this can prevent customers from using various MQ security controls and can make configuring some security scenarios more difficult.

The IBM MQ documentation relating to the use of AT-TLS has been updated here, and an slightly abbreviated version of this blog is available in the latest version of MP16, and finally don't forget to check out the MQ performance github repository for the latest performance reports spanning MQ for z/OS, MQ Appliance, MQ Distributed, MQ on OpenShift and MFT.

Permalink