Recently on a customer call I was asked “how can I tell what certificate my queue manager is using to identify itself with TLS?”. At first, I thought that the answer to this question was easy, just examine the queue manager’s configuration and keystore. But it slowly dawned on me that there’s a few gotcha’s, if’s, but’s and maybe’s that make answering this question a little more complicated.
In this post I’ll take you through some of these with the hopes to help you answer the question in the future for your own deployment of MQ.
There’s a couple of assumptions that I’m making here to try and simplify this blog post. I’ll be only talking about distributed platform queue managers acting as TLS servers and assuming a version of 9.2 (which at the time of writing is the latest). The version doesn’t matter too much currently as the last big change to TLS certificates and identity was back in version 8.0 of MQ where we added the multiple certificates feature and ability to declare what certificate to use by label in the queue manger configuration (more on this later). I’ve also only looked at distributed queue managers for giving examples of configuration/commands but the principles in here will apply to other platforms.
For the sake of brevity as well I’m only going to look at the cases where a queue manager acts as a TLS server as well. These are the channel types: SVRCONN, RCVR, SVR and the cluster variants of them.
The queue manager configuration
First let’s look at a simple case. Working out what certificate your queue manager configuration believes it is using.
For this example, the first step would be to look at the queue manager configuration, specifically two attributes on the qmgr:
- SSLKEYR = The keystore that the queue manager has been configured to use
- CERTLABL = The certificate the queue manager has been configured to use.
You can get both of those using runmqsc with the DISPLAY QMGR command:
DISPLAY QMGR SSLKEYR CERTLABL
Here you can see that my queue manager thinks the following:
- The keystore it has been configured to use is ‘/run/runmqserver/tls/key.kdb’
- Remember, IBM MQ automatically appends ‘.kdb’ onto the end of the keystore filename.
- The certificate it has been configured to use has the label ‘mycertificate’
Armed with this information I can use the IBM MQ certificate management tools to look at the certificate in that keystore to see what my queue manager has been configured to use:
runmqakm -cert -details -db /run/runmqserver/tls/key.kdb -stashed -label mycertificate
And with that we now can see the distinguished name (and other information) of the certificate that our queue manager has been configured to use. However, there is one issue with this approach. There is a window of time that the configuration of a queue manager and the internal representation of that configuration do not match up…
Refresh Security type(SSL)
If you make changes to your queue manager’s configuration or keystore contents then for those changes to be picked up, assuming that some TLS channels have already been started, you must execute the refresh security command.
This command will temporarily stop all inbound and outbound channels that have been enabled for TLS and refresh our internal TLS configuration so that the latest contents of the keystore and configuration settings are used before restarting all of those stopped channels. This means that if you do look at the configuration, but someone has come along and made changes but not executed the command, you could be looking at configuration that has not yet come into effect.
Of course, we’ve only focussed on one place that you can define a certificate label – the queue manager. However, the multiple certificates feature added the capability to allow you to use a different certificate on each channel. If all your channels have a blank CERTLABL setting, then any connections into your queue manager will use the certificate set on the queue manager level. However, if an application connects to a channel that has a CERTLABL set then we can use the same process above to lookup the certificate that will be returned with the same caveat that was mentioned above.
There is another caveat which means that an application connecting to a channel with a CERTLABL may not receive that certificate but instead the one set on the queue manager level. This is because of how multiple certificates works.
For an application to receive the TLS certificate defined on the channel, a special header needs to be sent by the application with the TLS Client hello. This header is called the Server Name Indication (SNI) header and is set automatically by IBM MQ applications. Normally the SNI header is set to the hostname of the server you are trying to connect to; However, by default, IBM MQ sets it to the channel name that you are trying to connect to. By setting this to the channel name, IBM MQ can work out which channel you are connecting to when it receives the TLS Client hello and so present the correct certificate. If we can’t work out the channel name (e.g. the SNI header is not present or correlates to a channel name that doesn’t exist or have a CERTLABL setting) then we will fall back to sending the queue manager’s certificate. So while your channel configuration may tell you what certificate it will use instead of the queue manager’s certificate; it is dependent on the application sending the correct SNI header to be able to do this.
While the default behaviour for most IBM MQ application types is to send the channel name as the SNI header, we released the OutboundSNI feature in version 9.2.1 which allows you to configure your applications to send either a hostname or channel name SNI header. Additionally, the ability to send the channel SNI is only available for C, Java, JMS, and unmanaged .NET applications types. Managed .NET, AMQP and MQXR applications are unable to send a channel name SNI header.
So how can I tell what certificate my queue manager is using right now?
In a perfect world, no changes would be made to our keystore or configuration without executing a timely REFRESH SECURITY command and any applications connecting to a channel with a CERTLABL set will send the correct SNI header so that multiple certificates can jump in and send the correct certificate. With this we could examine the MQ configuration and keystore to determine what certificate an application is likely to receive when it connects.
However, as shown above, this is not always the case. So, the question of “What certificate is my queue manager using?” seems to have an unsatisfactory answer of ‘It depends’. But, there is a way that we can determine what certificate a queue manager is using, and that is to ask the queue manager via any tool that will start a TLS handshake and print out the certificates that it receives when connected to the queue manager. For this my go-to choice is the `openssl s_client` tool.
Running the following command will open a TLS connection to the queue manager and then print out what certificate it receives.
openssl s_client -connect localhost:1414 -showcerts
I can even use the tool to send a SNI header and make sure it correctly points to a channel by following the rules detailed in this technote. https://www.ibm.com/support/pages/ibm-websphere-mq-how-does-mq-provide-multiple-certificates-certlabl-capability
openssl s_client -connect localhost:1414 -showcerts -servername rob2e-mq2e-channel.chl.mq.ibm.com
In this example I’ve connected to the same queue manager but given a SNI header with the aim to return the certificate set on ‘ROB.MQ.CHANNEL'. This channel name’s SNI header is: `rob2e-mq2e-channel.chl.mq.ibm.com`.
Note that this approach will generate AMQ9999E and AMQ9638E errors in your queue manager error logs as the queue manger does not like openssl connecting and disconnecting.
Hopefully this blog post has provided you the information you need to make sure you can accurately tell what certificate your queue manager is identifying with. If using configuration analysis or openssl queries to find out what certificate your queue manager is using for identification does not meet your requirements, then we would be really interested to hear this. I would encourage you to raise an RFE with your requirements or get in touch in the comments section of this blog post.