MQ

 View Only
Expand all | Collapse all

MQ Close and Disconnect methods hangs forever

  • 1.  MQ Close and Disconnect methods hangs forever

    Posted 16 days ago

    We are using a threading.timer call back method to reestablish connection when MQ goes down.
    The below call is never completing, it hangs forever. No exception or crash reported to the calling thread:

     LogMessage($"DisconnectMQ: Start closing all the MQ connections ", Tracing.LogCategory.Info, Tracing.LogLevel.NormalLow, null);
            //Close all MQ Queue connections
            try
            {
                if (queueIn != null && queueIn.IsOpen)
                {
                    LogMessage($"DisconnectMQ: queueIn is open calling queueIn.close() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
                    queueIn.Close();
                }
                if (queueIn_Delete != null && queueIn_Delete.IsOpen)
                {
                    LogMessage($"DisconnectMQ: queueIn_Delete is open calling queueOut.close() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
                    queueIn_Delete.Close();
                }
                if (queueOut != null && queueOut.IsOpen)
                {
                    LogMessage($"DisconnectMQ: queueOut is open calling queueOut.close() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
                    queueOut.Close();
                }
            }
            catch (Exception ex)
            {
                string message = "DisconnectMQ: Error Caught at CATCH 17, while closing the MQ connections";
                LogException(message, ex);
            }



    ------------------------------
    Balaji Patil
    ------------------------------


  • 2.  RE: MQ Close and Disconnect methods hangs forever

    IBM Champion
    Posted 15 days ago

    > We are using a threading.timer call back method to reestablish connection when MQ goes down.
    > The below call is never completing, it hangs forever.

    Well, looking at your code and reading your description, it looks like the code is waiting for the TCP timeout to happen. I think it is 10 minutes. Therefore, each method called in your code (IsOpen and Close) will have to go through the TCP timeout each time. Also, where is the disconnect call?

    Why are you re-inventing the wheel? Why don't you just let MQ reconnect to the queue manager?

    https://www.ibm.com/docs/en/ibm-mq/9.3?topic=programs-automatic-client-reconnection-in-net

    later

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London ON Canada
    https://capitalware.com
    ------------------------------

    Original Message


  • 3.  RE: MQ Close and Disconnect methods hangs forever

    Posted 14 days ago
    Edited by Balaji Patil 14 days ago

    Hi Roger,

    Thank you for your response.


    RE: "Why are you re-inventing the wheel? Why don't you just let MQ reconnect to the queue manager?"
          - We're already using the auto-reconnection logic to reconnect to the same queue manager (check the code snippet below), but due to the following reasons, I wrote the reconnection logic in my program.

    Reasons for writing reconnection logic on top of auto-reconnect logic:
    1. This is the part of upgrade project and the old system which is working since last 15 years has the same reconnection logic. So, my client wanted to have similar logic in the upgraded version.
    2. What if auto-reconnection logic failed to reconnect before timing out, how to handle such situation.
    3. The message volume is high (~100/sec) with tight cycle time. Messages will pileup if connection goes down even for a min. It will be very expensive if auto-reconnection takes to time to reconnect/fail.
    4. So, we thought of implementing this logic. 

    var connectionProperties = new Hashtable();
LogMessage($"ConnectMQ: Start creating MQ Connection(s)", Tracing.LogCategory.Info, Tracing.LogLevel.NormalLow, null);
connectionProperties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
connectionProperties.Add(MQC.HOST_NAME_PROPERTY, MQHostName);
connectionProperties.Add(MQC.PORT_PROPERTY, MQPort);
connectionProperties.Add(MQC.CHANNEL_PROPERTY, MQChannel);
//In case of network issues - reconnect to same queue manager
 if (!string.IsNullOrEmpty(MQHostName))
 {
 connectionProperties.Add(MQC.CONNECT_OPTIONS_PROPERTY, MQC.MQCNO_RECONNECT_Q_MGR);
 connectionProperties.Add(MQC.CONNECTION_NAME_PROPERTY, String.Format("{0}({1})", MQHostName, MQPort));
 }

LogMessage($"ConnectMQ: MQ Location=Remote. MQHostName = {MQHostName} MQPort = {MQPort} MQChannel = {MQChannel}", Tracing.LogCategory.Info, Tracing.LogLevel.NormalLow, null);
                
 //Start Creating Inbound connection for reading the messages.
if (MQQueueNameIn != null && MQQueueNameIn.Length > 0)
{
LogMessage($"ConnectMQ: Creating queueManagerIn connection", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugMedium, null);
 queueManagerIn = new MQQueueManager(MQManagerName, connectionProperties);

LogMessage($"ConnectMQ: Connecting to queueIn connection", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugMedium, null);
queueIn = queueManagerIn.AccessQueue(MQQueueNameIn, MQC.MQOO_BROWSE + MQC.MQOO_INPUT_SHARED + MQC.MQOO_FAIL_IF_QUIESCING + MQC.MQOO_INQUIRE);
}

    Thanks,

    Balaji

    ------------------------------
    Balaji Patil
    ------------------------------

    Original Message


  • 4.  RE: MQ Close and Disconnect methods hangs forever

    Posted 14 days ago

    If you are going to close the queue or the connection, why check the isopen method?

    Just close / disconnect if the object is not null and set it to null after the close / disconnect call. (use finally of try/catch of the close or disconnect).

    I believe the most time spent is in your isOpen check. You already know the connection is shot because you got a connection_broken so checking the isopen method is just a waste of time...

    Hope it helps



    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message


  • 5.  RE: MQ Close and Disconnect methods hangs forever

    IBM Champion
    Posted 14 days ago

    > I believe the most time spent is in your isOpen check. You already know the connection is shot because you got a connection_broken so checking the isopen method is just a waste of time...

    Exactly. Plus, how can an application be faster than MQ's own reconnection logic!

    connectionProperties.Add(MQC.CONNECT_OPTIONS_PROPERTY, MQC.MQCNO_RECONNECT_Q_MGR);

    Also, why does his code enable MQ reconnection but then want to use his own reconnection code? It is so weird. So, the MQ client library will reconnect, and then his code will force a reconnect; that is a scenario for a memory leak or connection leak.

    later

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London ON Canada
    https://capitalware.com
    ------------------------------

    Original Message


  • 6.  RE: MQ Close and Disconnect methods hangs forever

    Posted 13 days ago

    I believe his manual re-connection logic was meant to kick in after the MQ automated one had timed out...



    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message


  • 7.  RE: MQ Close and Disconnect methods hangs forever

    Posted 13 days ago

    Hi Francios, Roger:


    Thats exactly what we are trying to achieve. 
    If the auto-reconnection timed out, we want to execute the manual reconnection logic.
    So,

    1. How to know if the auto-reconnection timed out, will it throw an exception?
    2. What is the default timeout value, can we set the timeout through the code?

    As per your suggestion, I'll remove the following checking: 

     if (queueIn != null && queueIn.IsOpen)
  {
     LogMessage($"DisconnectMQ: queueIn is open calling queueIn.close() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
     queueIn.Close();
 }
 if (queueManagerIn != null && queueManagerIn.IsConnected)
   {
     LogMessage($"DisconnectMQ: queueManagerIn is open calling queueManagerIn.Disconnect() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
    queueManagerIn.Disconnect();
  }





    ------------------------------
    Balaji Patil
    ------------------------------

    Original Message


  • 8.  RE: MQ Close and Disconnect methods hangs forever

    Posted 13 days ago

    Replace it with:

    if (queueIn != null)
  {
     LogMessage($"DisconnectMQ: queueIn is open calling queueIn.close() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
try {
     queueIn.Close();
    } catch (Exception e) { 
    } finally {
      queueIn = null;
    }
 }
 if (queueManagerIn != null)
   {
     LogMessage($"DisconnectMQ: queueManagerIn is open calling queueManagerIn.Disconnect() ", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
     try {    
           queueManagerIn.Disconnect();
    } catch (Exception e) {
    } finally { 
           queueManagerIn = null;
    }
  }


    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message


  • 9.  RE: MQ Close and Disconnect methods hangs forever

    Posted 13 days ago

    Hi Francois: Thank you for the code snippet.

     Any suggestion on these:

    1. How to know if the auto-reconnection timed out, will it throw an exception?
    2. What is the default timeout value, can we set the timeout through the code?

     



    ------------------------------
    Balaji Patil
    ------------------------------

    Original Message


  • 10.  RE: MQ Close and Disconnect methods hangs forever

    Posted 13 days ago

    My expertise of the wrapper layers that provide higher level language support is very limited, however there are very few occasions where the underlying queue manager binding upon which this is all based could  appear to "hang forever". In particular, I'm unaware as to the threading model used and what state the HConn  was in when the "threading.timer call back method" ran. I  thought it might be worth pointing out that an hConn only supports serial MQI usage (see the description of MQCNO_HANDLE_SHARE_BLOCK and MQCNO_HANDLE_SHARE_NO_BLOCK. In the even that the option to block is selected then a call to MQCLOSE under a timer thread while the main worker thread was in a blocking MQI call (typically an MQGET with indefinite/long wait interval) would be  expected to appear to "hang forever".



    ------------------------------
    Andrew Hickson
    ------------------------------

    Original Message


  • 11.  RE: MQ Close and Disconnect methods hangs forever

    Posted 10 days ago

    Hi Andrew, Francois:

    Any suggestion on these:

    1. How to know if the auto-reconnection timed out, will it throw an exception?
    2. What is the default timeout value, can we set the timeout through the code?





    ------------------------------
    Balaji Patil
    ------------------------------

    Original Message


  • 12.  RE: MQ Close and Disconnect methods hangs forever

    Posted 10 days ago

    When you are in your logic that hangs forever, have you collected a Java thread dump to see exactly why your are hanging?

    One thing I have noticed with Java IBM MQ programming is you can encounter IBM MQ calls that hang forever in a TCP socket read. The below system property is not documented very well in the IBM MQ manual, but it handles breaking your IBM MQ call out of a stuck TCP socket read. The below example breaks out of the TCP socket read after 15 seconds. From my understanding, the default of a TCP socket read on Linux is to wait forever. This may be your underlying issue, but you may need to collect some more diagnostics to confirm this.

    -Dcom.ibm.mq.cfg.MQRCVBLKTO=15



    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message