Thanks to Sachin Tanna and Rob Parker for their reviews and feedback
Introduction
This article details how certificates are configured within IBM MQ and covers version 9.4.x on OpenShift. This is intended for MQ administrators, operations teams and security teams with a focus on MQ management. The article highlights key areas of certificate management including the creation and renewal of certificates.
This article assumes a basic knowledge of MQ objects, availability types and OpenShift objects e.g., PersistentVolumes
IBM MQ on OpenShift
Figure 1 – IBM MQ Objects
IBM MQ is an application that provides universal messaging backbone with robust connectivity for flexible and reliable messaging for disparate applications. It provides assured message queueing that decouples connecting applications availability.
In the OpenShift Operator a single CustomResource ‘QueueManager’ is used to define an MQ Container which includes an MQ server and a Liberty application. The MQ Server holds a ‘QueueManager’ which itself contains various objects needed by MQ such as queues, listeners and channels.
The Queue Manager might have multiple replicas for Native-HA and multi-instance availability models, and if using Native-HA the Queue Manager can also replicate data to another region using ‘Cross-Region Replication’.
A liberty server holds a java runtime for a Web User Interface and a REST Admin Interface that are accessible by administrative users and potentially automation tooling.
IBM MQ Certificates
Figure 2 - IBM MQ Certificate Usage illustrates the available certificate usages in the MQ Container.
Figure 2 - IBM MQ Certificate Usage
MQ Container
- There is an expectation that organisation specific certificates are used across the board, not relying on insecure self-signed certificates where provide
- WebUI and REST Admin keystore and truststore are defined in the mqwebuser.xml file by packaging a signed key and the certificate authority into keystore and truststore JKS files respectively and setting the path to their location in the ‘keystore’ and ‘truststore’ tags in the xml.
- The WebUI browser certificate is configured in the mqwebuser.xml file too, by setting the ‘keystore’ field with a path to the JKS that has a signed keystore certificate, and the clientAuthenticationSupported value to true in the ‘ssl’ tags.
This is part of the same keystore definition detailed for the WebUI and REST Admin. Note: MQ provides a self-signed certificate for the browser out of the box.
- IBM MQ Channels are the MQ object which is the ingress and egress point for the Queue Manager. It utilises keys inside the Queue Manager’s keystore and truststore for its channel security.
A channel (or Queue Manager level) definition defines a ‘certificate label’ which says which certificate to use from the keystore, and the ‘sslpeer’ which is used to verify a connecting applications certificates distinguished name. Channels are used by other Queue Manager channel connections and client connections which includes MQ Internet Pass-Thru (IPT).
In the ‘QueueManager’ custom resource - .spec.pki.trust.secret and .spec.pki.keys.secret are used to define OpenShift secrets containing the keystore and truststore certificates in a .pem format. These are built into the keystore database files for MQ.
- IBM MQ provides a managed file transfer facility called MQ MFT. This relies on having a designated Queue Manager specifically for MFT Agents.
MQ MFT requires a command and coordination queue manager which has bi-direction connections to Agent Queue Managers via standard MQ Channels which are secured in the same way.
- When using IBM MQ Native-HA, the 3 local replicas communicate log and data replication over their own communication channels. A separate cert/key must be provided for this communication in .pem format.
The same is true for Native-HA communication between Native-HA groups, a concept from 9.4 which is used for cross-region replication to define Live and Recovery groups.
This key is defined in the ‘QueueManager’ custom resource:
9.3.x or lower: .spec.queueManager.availability.tls
9.4.x or higher:
.spec.queueManager.availability.NativeHAGroups.local.tls (keystore)
.spec.queueManager.availability.nativeHAGroups.remotes.trust (truststore)
MQ Clients
- MQ Clients set the keystore and truststore in their code for example, java is set using javax.net.ssl.trustStore and javax.net.ssl.keyStore. These are used alongside the MQ Client libraries to connect to the Queue Manager channel
- Clients can also configure messages to be protected using Advanced Message Security – effectively encrypting a message at the client side of the channel connection. This means MQ only ever sees an encrypted message, enhancing security.
- On AIX, Linux, and Windows Clients, the location of the keystore database / certificates (JKS format) is provided in the keystore configuration file, which is keystore.conf by default and is pointed to within the Advanced Message Security Interceptors. Each Advanced Message Security user must have the keystore configuration file that points to a keystore file.
Additionally, use the runamscred command to protect sensitive information in that file.
- An MQ MFT Agent sits on a destination machine e.g., an SFTP server and send the file as a series of small MQ messages via an MQ Client connection.
All aspects of MFT (Queue Manager, Command, Coordination and Agents) use credentials, which are encrypted using a credentialKey (.key format) which is set in a JVM prooerty (-Dcom.ibm.wmqfte.cred.keyfile=/usr/home/credkeyfile.key), objectCredentialsKeyFile or set a common key using the commonCredentialsKeyFile in the installation.properties file.
Keystore and Truststore certificates and keys are defined in the (object) agent, coordination, command and logger .properties file by setting the objectSslPeerName, objectSslTrustStore, objectSslKeyStore which are in JKS or P12 format.
- There are two other agent types provided; Protocol Bridge Agent (direct access to files outside the MFT network) and Connect:Direct Agents (access to Sterling Connect:Direct networks). These have their own properties files configured similarly to a standard MQ MFT Agent where keystore, truststore and SSL Peer Name are set.
- MQ Internet Pass-Thru (IPT) runs as a stand-alone service that can receive and forward IBM MQ message flows, either between two IBM MQ queue managers or between an IBM MQ client and an IBM MQ queue manager. In its most basic form this is an MQ protocol forwarder, It receives connection requests from IBM MQ channels and listens on a TCP/IP port.
MQ IPT is configured to use an IPT parameter called ‘routes’ for its pass-thru which are defined in a mqipt.conf file. When a route has both SSLServer and SSLClient set, MQIPT accepts one incoming SSL/TLS secured connection and establishes a second SSL/TLS secured connection to another MQIPT or to a destination queue manager. The IBM MQ channel information is decrypted and re-encrypted between these two SSL/TLS connections.
MQ IPT can also be configured to be used in a ‘Proxy Mode’. In this mode, the route only forwards SSL/TLS data between two IBM MQ end-points; it does not participate in the SSL/TLS handshake and does not require any digital certificates.
Managing Certificates
Certificates are typically uploaded through the Queue Manager custom resource, meaning changes require an update to the OpenShift secret and a rollout of the Queue Manager instance.
This can either be automatic through changes to the custom resources, picked up by the operator and rolled out. Alternatively, pods deleted one by one will see the new configuration picked up by the Queue Manager.
Note: There will likely be a few seconds between the active Native-HA instance going down and a replica becoming the active instance.
References