MQ

 View Only
Expand all | Collapse all

Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

  • 1.  Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Tue March 31, 2020 08:38 AM
    Edited by Pavel Chuchma Tue March 31, 2020 10:33 AM

    MQQueue.Get() operation freezes for 5 minutes when getting a message from QueueManager using SSL on .NetCore 3.1. The same code works fine when running on .Net Framework (v4.8) or without SSL on both runtimes.

     

    Details:

    • Client: MQ Managed Client from NuGet (IBMMQDotnetClient/v9.1.4)
    • Server: docker from tag ibmcom/mq:9.1.4.0-r1 with basic SSL config using SSLCIPH(ANY_TLS12) SSLCAUTH(OPTIONAL)
    • Reproduced on: .Net Core 3.1.2 on Windows and .Net Core 3.1.1 on Linux
    • MQ Logs: MqGet-framework48.zipMqGet-core31.zip

    My Findings:

    • The attached MqGet-core31 log file tells:
      1. The client sends the get message request at 09:15:52.199588
      2. It receives response with message body almost immediately at 09:15:52.311050
      3. Then it calls another ReceiveOneTSH() to finish the get operation (probably confirming message get to server). It calls MQTCPConnection.Receive() and it freeze for 5 minutes to 09:20:52.412669 (until heartbeat timeout)
    • Possible reason:
      • MQTCPConnection keeps original socket and MQEncryptedSocket instance wrapping the same socket via SslStream
      • The method Receive() polls on the socket and then it calls MQEncryptedSocket.Read()
      • It looks like that data from the socket are read by SslStream in MQEncryptedSocket before the poll() method is called and poll() waits for its 5 minutes timeout. Or any other race condition around?
      • If I put a breakpoint before problematic poll() call and delay it for a moment, then it often passess without any delay!

    Test Code:
    This code finishes in ~2 seconds on .Net Framework, but it needs 5 minutes and ~2 seconds when it is run on .Net Core. (Win or Linux)

    Please, does anybody know a workaround or has a fix?

    // import server's CA
    MQConnectionPool.ImportServerCertificateAuthorityIfNotPresent(CertMqTestCa);

    Hashtable properties = new Hashtable {
        {MQC.HOST_NAME_PROPERTY, Mq1QmSslAnyTls12.Host},
        {MQC.PORT_PROPERTY, Mq1QmSslAnyTls12.Port},
        {MQC.CHANNEL_PROPERTY, Mq1QmSslAnyTls12.ChannelName},
        {MQC.USER_ID_PROPERTY, "admin"},
        {MQC.PASSWORD_PROPERTY, "changeit"},
        {MQC.CCSID_PROPERTY, 819},
        {MQC.SSL_CIPHER_SPEC_PROPERTY, "TLS_RSA_WITH_AES_256_CBC_SHA256"},
        {MQC.SSL_CERT_STORE_PROPERTY, "*USER"}
    };

    MQQueueManager queueManager;
    using (new MQCcsidSetter(properties)) {
        queueManager = new MQQueueManager(null, properties);
    }

    string requestText = Guid.NewGuid().ToString();
    string queueName = "SV.MqConnectionPoolTest";
    // put message - works fine
    using (MQQueue queue = queueManager.AccessQueue(queueName, MQC.MQOO_OUTPUT + MQC.MQOO_FAIL_IF_QUIESCING)) {
        byte[] requestBytes = Encoding.UTF8.GetBytes(requestText);
        MQMessage requestMessage = new MQMessage {Expiry = 3000, Priority = 4, CharacterSet = 1208, MessageType = MQC.MQMT_DATAGRAM};
        requestMessage.Write(requestBytes, 0, requestBytes.GetLength(0));
        queue.Put(requestMessage, new MQPutMessageOptions());
    }

    // get message back from the same queue
    using (MQQueue queue = queueManager.AccessQueue(queueName, MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_FAIL_IF_QUIESCING)) {
        while (true) {
            MQMessage msg = new MQMessage();
            queue.Get(msg, new MQGetMessageOptions()); // <<= !!!IT IS DELAYED HERE!!!
            msg.Seek(0);
            string msgContent = msg.ReadString(msg.MessageLength);
            if (requestText.Equals(msgContent)) {
                break;
            }
        }
    }
    Thread Dumps:
    1) MQ Client Receiver Thread
     SocketPal.Poll()
    Socket.Poll()
    MQTCPConnection.Receive()
    MQRcvThread.ReceiveBuffer()
    MQRcvThread.ReceiveOneTSH()
    MQRcvThread.Run()
    ThreadHelper.ThreadStart_Context()
    ExecutionContext.RunInternal()
    ThreadHelper.ThreadStart()
    Waits on Poll(), but it ends with 5 minute (heartbeat) timeout. Then it goes to this.network.Read() and immediately gets proper data.
    this.socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveTimeout, this.timeout);
    if (this.socket.Poll(this.timeout * 1000, SelectMode.SelectRead)) { // << WAITS HERE!
    length2 = this.network.Read(cBuffer, offset1, length1);
    2) Application Thread
    Monitor.Wait()
    MQSession.ExchangeTSH()
    MQProxyQueue.RequestMessages()
    MQProxyQueue.FlushQueue()
    MQProxyQueue.ProxyMQGET()
    MQFAP.zstMQGET()
    MQFAP.MQGET()
    MQDestination.Get()
    MQDestination.Get()
    MqConnectionPoolTest.TestMqDirect()
    Waits for entry.Reply to be filled by the receive thread:
    while (entry.Reply == null) {
    Monitor.Wait((object) entry, this.rmtReqEntMaxPollTime, true); // << WAITS HERE!
    ...
    }


  • 2.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    IBM Champion
    Posted Wed April 01, 2020 08:10 AM
    Edited by Francois Brandelik Wed April 01, 2020 08:14 AM
    Are you sure it is the get that makes you wait or the line that follows it?:
    queue.get(msg, new GetMessageOptions());
    msg.seek(0); //==>what is the purpose of this line?

    ------------------------------
    FJ Brandelik

    ------------------------------



  • 3.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Wed April 01, 2020 08:20 AM
    100% sure, it is inside the Get(), as it is visible from stack trace and MQ logs. Additionally, I run it in debugger ~10 times with the same result.

    ------------------------------
    Pavel Chuchma
    ------------------------------



  • 4.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    IBM Champion
    Posted Thu April 02, 2020 07:53 AM
    What is the value of share conversation on your server connection channel.
    Does the behavior change if you set this value to 1 ?

    ------------------------------
    FJ Brandelik

    ------------------------------



  • 5.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Thu April 02, 2020 08:32 AM

    FJ, thank you for your reply. I'm using default setting of official mq docker image. I didn't change any "share conversation" settings (I don't know it).

    Anyway, it looks that I found a workaround meanwhile. It is enough to use:

    queue.Get(msg, new MQGetMessageOptions {Options = MQC.MQGMO_WAIT, WaitInterval = 5000});

    instead of my original:

    queue.Get(msg, new MQGetMessageOptions());

    and it works fine. Value of WaitInterval is not relevant in this case (any value greater than 0 is fine) . It doesn't wait, because the message is already there (put by the first part of my test).
    This workaround is tested on .Net Core 3.1 running on Linux and Windows. It was freezing on both platforms before.

    It looks like a bug inside MQ Managed Client. The documentation says:
    If getMessageOptions is not specified, the message option used is MQGMO_NOWAIT.
    but NoWait waits for Heartbeat timeout on .Net Core.
    It I specify NoWait explicitly, 

    queue.Get(msg, new MQGetMessageOptions {Options = MQC.MQGMO_NO_WAIT});

    It doesn't work too.

    Please, does anybody know, how to report the bug to IBM developers? I probably don't have access to their support.
    Thanks.




    ------------------------------
    Pavel Chuchma
    ------------------------------



  • 6.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Fri January 15, 2021 03:22 PM
    I have the exact same issue with IBMMQDotnetClient 9.2.1 and .Net5. Created a case number with ibm support if they agree it is a bug.
    Thank you for the work around!

    Simon Frelier


    ------------------------------
    simon frelier
    it consultant
    ------------------------------



  • 7.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Tue January 19, 2021 12:52 PM
    Hello, 
    Just to mention we are having the same problem for some time now.
    Our setup was with .NETCore client v2.2. But it also happens with .NET Client v3.1.
    Only with .NetCore client (not with Framework). And only with SSL. 
    And with messages (16K+). In short, if the message is to be split in two, the client waits for getting the second part until the timeout.
    Status is now that IBM and Microsoft are discussing about the way how the poll() function is implemented in the MQ client.

    ------------------------------
    Jeroen Verbeiren
    ------------------------------



  • 8.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Tue January 19, 2021 12:52 PM
    Hello,
    We are having a similar issue (for several time now) using MQDotNetClient 9.1.3 using .NETCore 2.2. 
    It happens with messages larger than 16k and using SSL. If the message is to be split up in different segments, the client waits until timeout to get the 2nd (and other) segments of the message. It doesn't happen with .NET framework. It doesn't happen if no SSL is used.
    It also happens with .NETCore 3.1.
    A case is open with IBM. They have been able to replicate the issue.
    They are working/discussing now with Microsoft about the implementation of the poll() function in .NET and MQClient.
    It looks like we're getting closer to a solution, but we're not there yet.
    Jeroen Verbeiren

    ------------------------------
    Jeroen Verbeiren
    ------------------------------



  • 9.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Sun October 17, 2021 08:06 AM
    Make sure your using a MQ server higher or equal to 9.1.4 or use an ibm client higher then 9.1.6. IBM provided me a fix for client 9.1.4 but I decided to wait it out till the fix was generaly available in a normal release.

    ------------------------------
    simon frelier
    it consultant
    ------------------------------



  • 10.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Mon October 18, 2021 03:45 AM
    Hello,
    Apologies for not keeping this post up to date.
    A fix has been indeed implemented on MQ client 9.2.0.1
    But I am also still waiting to have it generally available (for several months now)...
    best regards,
    jeroen

    ------------------------------
    Jeroen Verbeiren
    ------------------------------



  • 11.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Thu November 18, 2021 06:23 AM

    Yup, Works with 9.2.0.1



    ------------------------------
    Pareek Ashutosh
    ------------------------------



  • 12.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Mon October 18, 2021 10:06 AM
    Any updates on this issue from IBM? Is there an apar for this?
    I have the same problem when using the async messageconsumer in the xms-library, running .net core 3.1 mq 9.2.0 and the latest client from nuget-store(9.2.3).
    Looking at the MS documentation there is some issues using the poll under ssl Time-out occurs when an application calls the Poll method - .NET Framework
    @simon frelier you mention an fix, have you tried it? 







    ------------------------------
    Daniel Nordkvist
    ------------------------------



  • 13.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Wed October 20, 2021 02:33 AM
    Hi,

    APAR IJ20591. But still in review (since March 2021). :-(
    There were calls between IBM and Microsoft. It had something todo with how the .NET client in MQ reads data from the socket when using SSL.
    So quite low-level.

    best regards,


    ------------------------------
    Jeroen Verbeiren
    ------------------------------



  • 14.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Wed October 20, 2021 08:56 AM
    Hi,
    When you mention a fix for 9.2.0.1. Has it been delivered to you as an ifix? Have you tried it?
    I'm still waiting for an answer from the "ibm lab" as they call it. 

    Another question is why the client has to be released at the ibm-mq releasedates. Takes ages to get things solved.

    Regards



    ------------------------------
    Daniel Nordkvist
    ------------------------------



  • 15.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Thu November 18, 2021 06:23 AM
    We are also facing the exact same issue. .Net Core 3.1 + SSL + Message Size >16KB and application takes about 5min to read the message. Exact same code when targeted to .Net FW- 4.7, works just fine.
    Any update on the fix of this issue? Any workaround?

    ------------------------------
    Pareek Ashutosh
    ------------------------------



  • 16.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Fri November 19, 2021 02:54 AM
    Hi
    I have received an ifix from IBM on this issue. There is an apar but the fix is not generally available.

    (I dont think there is a workaround on this issue. But you could lower the heartbeat interval on the server connection channel and you will see it follows that setting, default is 300 seconds. For sure not a good or recommended workaround.)
    Regards
    Daniel

    ------------------------------
    Daniel Nordkvist
    ------------------------------



  • 17.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Thu November 18, 2021 11:05 AM
    9.2.4 was released today but I am still facing this with latest version. Anyone knows whats the update on this issue?

    ------------------------------
    Pareek Ashutosh
    ------------------------------



  • 18.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Mon November 22, 2021 02:32 AM

    The latest update is that APARs  IT36749 and IJ2020591 have been resolved. AFAIK it is as of yet not ascertained how/when they are to be delivered (CD release/LTS FP). However by contacting IBM Support you should be able to receive an ifix in the mean time.

    As was previously said in this thread, specifying a high (the value must be higher than the actual time it takes to retrieve the message) wait interval could/would work as a circumvention. Word is however that it does not always work (even though it appears to do it most of the time). The proper way forward it to ask IBM Support for an iFix and also to ask how/when it is expected to be included in the product.



    ------------------------------
    Johan Hellström
    ------------------------------



  • 19.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Wed December 29, 2021 12:11 AM
    There is an update on this issue. APAR is now closed and will be part of 9.2.0.5.
    IJ20591: Managed .NET SSL application making MQGET calls unexpectedly receives MQRC_CONNECTION_BROKEN when running in .NET Core (ibm.com)

    ------------------------------
    Pareek Ashutosh
    ------------------------------



  • 20.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Tue October 25, 2022 11:01 AM

    I am still facing the issue. I am using MQ 9.2.1 and .Net Core 3.1. Is there any work-around? We are stuck after upgrading the application to .net core stack.



    ------------------------------
    Abhishek Joshi
    ------------------------------



  • 21.  RE: Managed IBM MQ .Net Client v9.1.4 freezes on get via SSL when running on .Net Core 3.1

    Posted Tue October 25, 2022 11:36 AM

    If you see the "fix is targeted" section of APAR IJ20591, it is mentioned as

    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v9.2 LTS   9.2.0.5
    v9.x CD    9.3.0.0

    Hence you need to use a version higher or equal to IBM MQ v9.2.0.5 or IBM MQ v9.3



    ------------------------------
    Ram Subba Rao Chalamalasetti
    ------------------------------