Original Message:
Sent: Mon October 18, 2021 03:44 AM
From: Jeroen Verbeiren
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
Hello,
Apologies for not keeping this post up to date.
A fix has been indeed implemented on MQ client 9.2.0.1
But I am also still waiting to have it generally available (for several months now)...
best regards,
jeroen
------------------------------
Jeroen Verbeiren
Original Message:
Sent: Sun October 17, 2021 08:06 AM
From: simon frelier
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
Make sure your using a MQ server higher or equal to 9.1.4 or use an ibm client higher then 9.1.6. IBM provided me a fix for client 9.1.4 but I decided to wait it out till the fix was generaly available in a normal release.
------------------------------
simon frelier
it consultant
Original Message:
Sent: Tue January 19, 2021 10:38 AM
From: Jeroen Verbeiren
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
Hello,
We are having a similar issue (for several time now) using MQDotNetClient 9.1.3 using .NETCore 2.2.
It happens with messages larger than 16k and using SSL. If the message is to be split up in different segments, the client waits until timeout to get the 2nd (and other) segments of the message. It doesn't happen with .NET framework. It doesn't happen if no SSL is used.
It also happens with .NETCore 3.1.
A case is open with IBM. They have been able to replicate the issue.
They are working/discussing now with Microsoft about the implementation of the poll() function in .NET and MQClient.
It looks like we're getting closer to a solution, but we're not there yet.
Jeroen Verbeiren
------------------------------
Jeroen Verbeiren
Original Message:
Sent: Fri January 15, 2021 06:38 AM
From: simon frelier
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
I have the exact same issue with IBMMQDotnetClient 9.2.1 and .Net5. Created a case number with ibm support if they agree it is a bug.
Thank you for the work around!
Simon Frelier
------------------------------
simon frelier
it consultant
Original Message:
Sent: Thu April 02, 2020 08:32 AM
From: Pavel Chuchma
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
FJ, thank you for your reply. I'm using default setting of official mq docker image. I didn't change any "share conversation" settings (I don't know it).
Anyway, it looks that I found a workaround meanwhile. It is enough to use:
queue.Get(msg, new MQGetMessageOptions {Options = MQC.MQGMO_WAIT, WaitInterval = 5000});
instead of my original:
queue.Get(msg, new MQGetMessageOptions());
and it works fine. Value of WaitInterval is not relevant in this case (any value greater than 0 is fine) . It doesn't wait, because the message is already there (put by the first part of my test).
This workaround is tested on .Net Core 3.1 running on Linux and Windows. It was freezing on both platforms before.
It looks like a bug inside MQ Managed Client. The documentation says:
If getMessageOptions is not specified, the message option used is MQGMO_NOWAIT.
but NoWait waits for Heartbeat timeout on .Net Core.
It I specify NoWait explicitly,
queue.Get(msg, new MQGetMessageOptions {Options = MQC.MQGMO_NO_WAIT});
It doesn't work too.
Please, does anybody know, how to report the bug to IBM developers? I probably don't have access to their support.
Thanks.
------------------------------
Pavel Chuchma
Original Message:
Sent: Thu April 02, 2020 07:53 AM
From: Francois Brandelik
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
What is the value of share conversation on your server connection channel.
Does the behavior change if you set this value to 1 ?
------------------------------
FJ Brandelik
Original Message:
Sent: Wed April 01, 2020 08:20 AM
From: Pavel Chuchma
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
100% sure, it is inside the Get(), as it is visible from stack trace and MQ logs. Additionally, I run it in debugger ~10 times with the same result.
------------------------------
Pavel Chuchma
Original Message:
Sent: Wed April 01, 2020 08:09 AM
From: Francois Brandelik
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
Are you sure it is the get that makes you wait or the line that follows it?:queue.get(msg, new GetMessageOptions());
msg.seek(0); //==>what is the purpose of this line?
------------------------------
FJ Brandelik
Original Message:
Sent: Mon March 30, 2020 11:00 AM
From: Pavel Chuchma
Subject: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1
MQQueue.Get() operation freezes for 5 minutes when getting a message from QueueManager using SSL on .NetCore 3.1. The same code works fine when running on .Net Framework (v4.8) or without SSL on both runtimes.
Details:
- Client: MQ Managed Client from NuGet (IBMMQDotnetClient/v9.1.4)
- Server: docker from tag ibmcom/mq:9.1.4.0-r1 with basic SSL config using SSLCIPH(ANY_TLS12) SSLCAUTH(OPTIONAL)
- Reproduced on: .Net Core 3.1.2 on Windows and .Net Core 3.1.1 on Linux
- MQ Logs: MqGet-framework48.zip, MqGet-core31.zip
My Findings:
- The attached MqGet-core31 log file tells:
- The client sends the get message request at 09:15:52.199588
- It receives response with message body almost immediately at 09:15:52.311050
- Then it calls another ReceiveOneTSH() to finish the get operation (probably confirming message get to server). It calls MQTCPConnection.Receive() and it freeze for 5 minutes to 09:20:52.412669 (until heartbeat timeout)
- Possible reason:
- MQTCPConnection keeps original socket and MQEncryptedSocket instance wrapping the same socket via SslStream
- The method Receive() polls on the socket and then it calls MQEncryptedSocket.Read()
- It looks like that data from the socket are read by SslStream in MQEncryptedSocket before the poll() method is called and poll() waits for its 5 minutes timeout. Or any other race condition around?
- If I put a breakpoint before problematic poll() call and delay it for a moment, then it often passess without any delay!
Test Code:
This code finishes in ~2 seconds on .Net Framework, but it needs 5 minutes and ~2 seconds when it is run on .Net Core. (Win or Linux)
Please, does anybody know a workaround or has a fix?
// import server's CA
MQConnectionPool.ImportServerCertificateAuthorityIfNotPresent(CertMqTestCa);
Hashtable properties = new Hashtable {
{MQC.HOST_NAME_PROPERTY, Mq1QmSslAnyTls12.Host},
{MQC.PORT_PROPERTY, Mq1QmSslAnyTls12.Port},
{MQC.CHANNEL_PROPERTY, Mq1QmSslAnyTls12.ChannelName},
{MQC.USER_ID_PROPERTY, "admin"},
{MQC.PASSWORD_PROPERTY, "changeit"},
{MQC.CCSID_PROPERTY, 819},
{MQC.SSL_CIPHER_SPEC_PROPERTY, "TLS_RSA_WITH_AES_256_CBC_SHA256"},
{MQC.SSL_CERT_STORE_PROPERTY, "*USER"}
};
MQQueueManager queueManager;
using (new MQCcsidSetter(properties)) {
queueManager = new MQQueueManager(null, properties);
}
string requestText = Guid.NewGuid().ToString();
string queueName = "SV.MqConnectionPoolTest";
// put message - works fine
using (MQQueue queue = queueManager.AccessQueue(queueName, MQC.MQOO_OUTPUT + MQC.MQOO_FAIL_IF_QUIESCING)) {
byte[] requestBytes = Encoding.UTF8.GetBytes(requestText);
MQMessage requestMessage = new MQMessage {Expiry = 3000, Priority = 4, CharacterSet = 1208, MessageType = MQC.MQMT_DATAGRAM};
requestMessage.Write(requestBytes, 0, requestBytes.GetLength(0));
queue.Put(requestMessage, new MQPutMessageOptions());
}
// get message back from the same queue
using (MQQueue queue = queueManager.AccessQueue(queueName, MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_FAIL_IF_QUIESCING)) {
while (true) {
MQMessage msg = new MQMessage();
queue.Get(msg, new MQGetMessageOptions()); // <<= !!!IT IS DELAYED HERE!!!
msg.Seek(0);
string msgContent = msg.ReadString(msg.MessageLength);
if (requestText.Equals(msgContent)) {
break;
}
}
}
Thread Dumps:
1) MQ Client Receiver Thread
SocketPal.Poll()
Socket.Poll()
MQTCPConnection.Receive()
MQRcvThread.ReceiveBuffer()
MQRcvThread.ReceiveOneTSH()
MQRcvThread.Run()
ThreadHelper.ThreadStart_Context()
ExecutionContext.RunInternal()
ThreadHelper.ThreadStart()
Waits on Poll(), but it ends with 5 minute (heartbeat) timeout. Then it goes to this.network.Read() and immediately gets proper data.
this.socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveTimeout, this.timeout);
if (this.socket.Poll(this.timeout * 1000, SelectMode.SelectRead)) { // << WAITS HERE!
length2 = this.network.Read(cBuffer, offset1, length1);
2) Application Thread
Monitor.Wait()
MQSession.ExchangeTSH()
MQProxyQueue.RequestMessages()
MQProxyQueue.FlushQueue()
MQProxyQueue.ProxyMQGET()
MQFAP.zstMQGET()
MQFAP.MQGET()
MQDestination.Get()
MQDestination.Get()
MqConnectionPoolTest.TestMqDirect()
Waits for entry.Reply to be filled by the receive thread:while (entry.Reply == null) {
Monitor.Wait((object) entry, this.rmtReqEntMaxPollTime, true); // << WAITS HERE!
...
}