And by configuring an SSLPEER for my MQ Client that purposely did not match the cert being presented by the queue manager, that too was able to produce the AMQ9665E error in the QM error log with ???? for the channel name. This is what Tim reported. So there are at least 2 scenarios where the client side causes the TLS handshake to terminate before the client channel name is know to the queue manager.
Can anyone think of any others? Knowing these is helpful to the MQ Admin trying to solve a TLS issue when they don't have access to the client side logs and the queue manager side logs are only saying the client terminated the connection but I can't tell you the channel name.
Original Message:
Sent: Thu March 16, 2023 06:09 PM
From: Peter Potkay
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
I was able to reproduce the AMQ9665E error with ???? for the channel name.
On my client machine I created a new empty keystore.
I created a JSON CCDT that specified ANY_TLS12_OR_HIGHER for the Cipher Suite and a value for SSLPEERNAME that matched what the QM cert should have for the DN field.
Opened a command prompt.
SET MQSSLKEYR=C:\MyCoolStuff\key
SET MQCHLLIB=C:\MyCoolStuff\
SET MQCHLTAB=MyCoolCCDT.json
amqscnxc MyCoolQueueManager
MQCONNX ended with reason code 2393
mqrc 2393 0x00000959 MQRC_SSL_INITIALIZATION_ERROR
In the Queue Manager's log, the AMQ9665E error with ???? for the channel name.
I then populate the empty keystore with the Root Certificate for the Certificate Authority that signed the queue manager's cert's Intermediate cert and the connection works.
OK, so it does not matter if its Java or C based. Certain TLS handshake errors occur before the channel name flows and others occur after the channel name is known. My use case shows a missing CA cert on the client side will cause AMQ9665E error with ???? for the channel name. Tim just posed that an SSLPEERNAME failure on the client side will do the same.
If I configure nothing TLS related on the client side and hit a TLS enabled SVRCONN channel, the queue manager does record the client channel name and AMQ9639E: Remote channel 'ChannelNameHere' did not specify a CipherSpec. Hmmmm, this must be because there is no TLS handshake occurring at all, so the channel name is known right away and can be added to the error log.
Back to my problem business app, after much back and forth they confirmed they added the Root Certificate to their trust file and they are finally working.
------------------------------
Peter Potkay
Original Message:
Sent: Thu March 16, 2023 03:25 PM
From: Tim Zielke
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
I was actually doing some IBM MQ TLS certificate maintenance today, and I noticed that AMQ9665E messages were being generated for cases where the TLS handshake was fine. They were generated between the time of updating the queue manager certificate and then going to the client server (which takes me a few minutes) to update the ccdt.json to update the new serialnumber for the ssl peer check. So this is a use case where AMQ9665E is producing a misleading message.
This is the mrqc doc for AMQ9665E.
mqrc AMQ9665
536909413 0x20009665 rrcE_SSL_SOCKET_CLOSED
MESSAGE:
SSL connection closed by remote end of channel '<insert one>'.
EXPLANATION:
The SSL or TLS connection was closed by the remote host '<insert three>' during
the secure socket handshake. The channel is '<insert one>'; in some cases its
name cannot be determined and so is shown as '????'. The channel did not start.
ACTION:
Check the remote end of the channel for SSL and TLS errors. Fix them and
restart the channel.
In my case, it was not a TLS handshake failure, but an SSLPEER check failure. So just something to be aware of when investigating these AMQ9665E errors.
------------------------------
Tim Zielke
Original Message:
Sent: Wed March 15, 2023 03:53 PM
From: Tim Zielke
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
I would think a C or Java client would cause the same behavior if they break the TLS handshake. I am pretty sure the TLS handshake happens before the IBM MQ client side can send any data to the queue manager, so all the queue manager knows is a TLS handshake failed. Morag mentioned that more recent versions of the IBM MQ client can leverage the SNI in the TLS handshake to provide a channel name to the queue manager, so maybe that AMQ9665 error will be more descriptive (provide a channel name) in the future, or maybe it currently does. Not sure.
The most likely issue here is that the IBM MQ client side can not access its keystore (wrong file name), the keystore is missing the root CA, or something along those lines. The JSSE trace should explain the root cause of the issue.
------------------------------
Tim Zielke
Original Message:
Sent: Wed March 15, 2023 03:06 PM
From: Peter Potkay
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
I had forgotten about that excellent article, thanks Tim. I'm gonna lean on the app team to read it and offer to help them review the results of the trace.
So I'm inferring that the errors that are seen in the MQ queue manager's log will be different based on whether the client is C based or Java based, even for the exact same root cause on the client side. True?
And is it true that at least for a Java based client, there is some TLS back and forth between the client and the server (queue manager) before the channel name is ever shared? If true and the TLS handshake is ended by the client in this early phase, that is why the queue manager can only report the client ended the conversation for some TLS reason and we don't yet have a channel name to log.
My prime suspicion at this point is they are using a trust store that does not have Root Certificate from the Cert Authority that signed the queue manager's cert. Assuming this is true, and they are JMS, and they are using a non IBM JRE, is that why they are ending the TLS conversation and its happening before the MQ CLNTCONN/SVRCONN value is sent down the wire?
------------------------------
Peter Potkay
Original Message:
Sent: Wed March 15, 2023 01:25 PM
From: Tim Zielke
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
This blog post is helpful for being able to read the -Djavax.net.debug=all trace.
https://community.ibm.com/community/user/integration/blogs/tim-zielke1/2020/06/10/java-jsse-debug-trace
From what I have experienced, you get the AMQ9665E when the TLS handshake is broken by the client. I don't believe the TLS handshake that the queue manager uses considers the client authentication as required, based on my experience/understanding. So for the client side to break the TLS handshake, it is usually something required in the TLS handshake, like the client side needing to trust the queue manager certificate.
------------------------------
Tim Zielke
Original Message:
Sent: Wed March 15, 2023 01:05 PM
From: Francois Brandelik
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
Running their program (appserver, stand alone etc) with the debug flag for javax.net will give in the output the details of which keystore / truststore is being used, which cert is being used for the exchange etc... It is a quite detailed output.
What the above link you provided did not show is: running correctly but using the wrong keystore / truststore.
My guess is that you are using JMS from within an app server. The MQ client team needs to talk to the app team to find out how to determine which keystore is being used for the JMS connection and add to that key store / truststore the right certs.
Also the client keystore should have a certificate labelled ibmwebspheremqsomeid .
Hope this helps
------------------------------
Francois Brandelik
Original Message:
Sent: Wed March 15, 2023 11:37 AM
From: Peter Potkay
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
If I could only get valid info from the client side this would be quickly solved. Between Event messages, App Activity Trace and the Queue Manager error logs IBM MQ is usually very good providing me the info I need on the server side to be able to tell the MQ Client what they are doing wrong, instead of asking, pleading, begging for info from the app and never really trusting what I hear anyway.
Trying to pull info from the app team has been frustrating to say the least. I just can't figure out how to recreate these error messages myself to get a clue what they are doing wrong. How do I attempt to connect as an MQ Client to a queue manager where the channel is TLS (not mTLS) and cause the connection to abort so that the Queue Manager records the channel as ???? and the error simply says "The SSL or TLS connection was closed by the remote host during the secure socket handshake."
I tried amqscnxc from a command window with zero TLS set up. No keystore referenced, no CCDT with CipherSpec, and still the MQ Queue Manager was able to tell me the channel name I attempted, and tell me the client did not provide a Cipher Spec. So, any ideas what I do as an MQ Client that causes the TLS handshake to terminate, to terminate by the client, after a network connection is established to the Queue Manager but before the channel name is sent?
I continue to try and get relevant info from the actual app support team. The application does appear to be JMS. I got the below snippet from them. I shared this URL with them asking if they see additional info in their logs similar to the examples in this URL: https://www.ibm.com/docs/en/ibm-mq/9.2?topic=problems-tls-troubleshooting
ERROR 43 --- [ntContainer#0-1] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'APP.QUEUE.NAME.HERE' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name 'Client'.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').
------------------------------
Peter Potkay
Original Message:
Sent: Wed March 15, 2023 08:35 AM
From: Francois Brandelik
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
Peter,
Have you tried with a Java app or JMS app instead of a C app?
Apart from the differences in the cipherspec, that can be somewhat averted by using a ccdt, I would make sure that the java client uses debug on startup for the SSL part. In my experience these errors come mostly because the actual keystore / truststore used is not the one the app owner thinks is being used.
This would be shown in detail if java was started with either of the following command line examples
java -Djavax.net.debug=ssljava -Djavax.net.debug=all
Hope it helps
------------------------------
Francois Brandelik
Original Message:
Sent: Tue March 14, 2023 01:59 PM
From: Peter Potkay
Subject: How do I force AMQ9665E: SSL connection closed by remote end of channel '????'.
I see occurrence's of AMQ9665E and AMQ9999E where the channel name is recorded in the MQ error logs as '????'. Below is one example of this pair of errors. I changed the IP address to quad 255 to protect the guilty.
The app client is clearly struggling with making a successful TLS connection. I am pretty sure we got the right client app people scheduled for a call later today to sort this out. What I am struggling with is exactly what are they doing wrong on their end that would cause this error on the MQ server side where the client channel name is ????
I want to be able to recreate it so I can give better advice. If I try to connect to that channel using sample MQ programs (e.g. amqscnxc) and purposely avoid configuring anything TLS related on my client, the error on the MQ server side does show the channel name and does mention the remote side (me) is not specifying a Cipher. OK, a very useful error. But the errors these guys are generating are puzzling. What exactly does a client have to do wrong to produce a pair of error messages in the QM error log like below?
03/14/2023 01:37:25 PM - Process(999999.9999) User(mqm) Program(amqrmppa)
Host(MyAwesomeServer) Installation(Installation1)
VRMF() QMgr(MyAwesomeQmgr)
Time(2023-03-14T17:37:25.527Z)
RemoteHost(255.255.255.255)
ArithInsert1(420)
CommentInsert1(????)
CommentInsert2(gsk_secure_soc_init)
CommentInsert3(255.255.255.255)
AMQ9665E: SSL connection closed by remote end of channel '????'.
EXPLANATION:
The SSL or TLS connection was closed by the remote host '255.255.255.255' during
the secure socket handshake. The channel is '????'; in some cases its name
cannot be determined and so is shown as '????'. The channel did not start.
ACTION:
Check the remote end of the channel for SSL and TLS errors. Fix them and
restart the channel.
03/14/2023 01:37:25 PM - Process(999999.9999) User(mqm) Program(amqrmppa)
Host(MyAwesomeServer) Installation(Installation1)
VRMF() QMgr(MyAwesomeQmgr)
Time(2023-03-14T17:37:25.527Z)
CommentInsert1(????)
CommentInsert2(140789)
CommentInsert3(255.255.255.255)
AMQ9999E: Channel '????' to host '255.255.255.255' ended abnormally.
EXPLANATION:
The channel program running under process ID 140789 for channel '????' ended
abnormally. The host name is '255.255.255.255'; in some cases the host name
cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
------------------------------
Peter Potkay
------------------------------