MQ

MQ

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

MQ Stale connections

  • 1.  MQ Stale connections

    Posted Mon April 03, 2017 03:59 PM

    Hi All, requesting you suggestions to resolve below issue -  

    XPosting from Application Infra Forum https://www.imwuc.org/p/fo/st/thread=4966

    In our Core APIs implementation, we have multiple activity services running in WAS v8.5.5.11 and using MQ v7.5 for APIs. These activity services (total 31) uses Spring 4.1 (26) & MDB Framework (5). Each Activity service has own QCF.

    We are observing that number of connections in MQ keep on increasing over a period of time of 2 to 3 days and WAS is not able to make any new connections with MQ QMgr once the total number of connections crosses max limit 2048.

    Once the max connection reaches, it forces the WAS to restart. Post WAS restart, connections are reset however connection counts (CHSTATUS = NONE) keep on increasing slowly and not coming down once load it reduced during non business hours or weekend.

    Another observation is, once the WAS is restarted, exactly after 24 hours, we find unusual increase in stale connection buildup which doesn’t come down as we mentioned earlier.

    As per the analysis, we can see that there are 3 types of objects opened by the connections:

     

    After Server Restart:

    Total Number of channel instances: 36

    Total shared conversations: 194

    Unique CONNs: 194

    Objects opened by CONNs:

        129 ""

          1 "QMGR"

         66 "QUEUE

     

    After 3 days of continuous running:

    Total Number of channel instances: 107

    Total shared conversations: 414

    Unique CONNs: 413

    Objects opened by CONNs:

        112 ""

        241 "QMGR"

         60 "QUEUE"

     

    "" and "QUEUE" object type connections remain almost same over period of time but "QMGR" type connections keep on increasing and it seems they are closed only after a server restart.

    Can you help in understanding why these "QMGR" type connections are getting created and if they are required how can we ensure they are closed regularly?

     Have you faced similar issues and observations & can help us resolve?

    Thanks!



  • 2.  RE: MQ Stale connections

    Posted Mon April 03, 2017 04:49 PM

    May I ask a few questions to more fully understand your question?

    You say "connection counts (CHSTATUS = NONE)". What do you mean by this?

    What command are you issuing to get your totals? Is it DISPLAY CONN? Can you provide an example of the "" object type using DISPLAY CONN.

    Are the extra QMGR type object handles associated with a channel? Again you can see this in DISPLAY CONN. Are they from a expected source?

    Look at the CHSTATUS of the additional channel instances (DISPLAY CHSTATUS), are they doing anything? For example, does the number of bytes moved over the channel keep increasing? What is the SUBSTATE? Maybe they are stale connections.



  • 3.  RE: MQ Stale connections

    Posted Tue April 04, 2017 07:28 AM
      |   view attached

    Thanks Morag for your attention.

    Here are scripts and refer to graphs in attachment.

    #!/bin/ksh

    CHL=${1}
    QMGR=${2}
    echo "Total Number of channel instances: $(echo "DIS CHS(${CHL}) ALL"|runmqsc $QMGR| grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }'|wc -l)"
    echo "Total shared conversations: $(expr $(echo $(echo "DIS CHS(${CHL}) ALL"|runmqsc $QMGR| grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }'|awk -F, '{print $2}')|sed -e 's/ / + /g'))"
    echo "Unique CONNs: $(echo "dis conn(*) type(all) where(channel eq ${CHL})"|runmqsc $QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'|awk -F, '{print $1}'|sort -u|wc -l)"
    echo "Objects opened by CONNs:"
    echo "dis conn(*) type(all) where(channel eq ${CHL})"|runmqsc $QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'|awk -F, '{print $5}'|sort|uniq -c



  • 4.  RE: MQ Stale connections

    Posted Tue April 04, 2017 10:43 PM

    You have not answered most of my questions. Please can you read them again and try?

    I can see from your script that you don't check to see if there is any object handle before printing out OBJTYPE, which means that your OBJTYPE of "" is just a connection that hasn't opened any objects.



  • 5.  RE: MQ Stale connections

    Posted Wed April 05, 2017 07:28 AM

    In Reply to Morag Hughson:

    May I ask a few questions to more fully understand your question?

    Let me try here.

    You say "connection counts (CHSTATUS = NONE)". What do you mean by this?

    Sorry, my bad, ts ASTATE(NONE) for the connection. Command used for listing the connections for specific channel used- 

    echo "dis conn(*) type(all) where(channel eq CHANNELNAME)"|runmqsc QMGRNAME|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'

    Observations - Below type are increasing after24 Hrs :-(

    E07DDE5801B4CE22","CHANNELNAME","IP","QMGRNAME","QMGR","NONE","WebSphere MQ Channel","WebSphere MQ Client for Java","MQOO_INQUIRE

    What command are you issuing to get your totals? Is it DISPLAY CONN? Can you provide an example of the "" object type using DISPLAY CONN.

    Repeating from my earlier post. Below is command for Channel instances-

    CHL=${1}
    QMGR=${2}
    echo "Total Number of channel instances: $(echo "DIS CHS(${CHL}) ALL"|runmqsc $QMGR| grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }'|wc -l)"
    echo "Total shared conversations: $(expr $(echo $(echo "DIS CHS(${CHL}) ALL"|runmqsc $QMGR| grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }'|awk -F, '{print $2}')|sed -e 's/ / + /g'))"
    echo "Unique CONNs: $(echo "dis conn(*) type(all) where(channel eq ${CHL})"|runmqsc $QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'|awk -F, '{print $1}'|sort -u|wc -l)"
    echo "Objects opened by CONNs:"
    echo "dis conn(*) type(all) where(channel eq ${CHL})"|runmqsc $QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'|awk -F, '{print $5}'|sort|uniq -c

    Observations (After 3 days of continuous running):

    Total Number of channel instances: 107

    Total shared conversations: 414

    Unique CONNs: 413

    Objects opened by CONNs:

        112 ""

        241 "QMGR"

         60 "QUEUE"

     

    "" and "QUEUE" object type connections remain almost same over period of time but "QMGR" type connections keep on increasing and it seems they are closed only after a server restart.

     

    Are the extra QMGR type object handles associated with a channel? Again you can see this in DISPLAY CONN. Are they from a expected source?

    Yes, extra are there as explained above but they are from expected source as per IP verification.

    Look at the CHSTATUS of the additional channel instances (DISPLAY CHSTATUS), are they doing anything? For example, does the number of bytes moved over the channel keep increasing? What is the SUBSTATE? Maybe they are stale connections.

    Yes, bytes are increasing. All Substate are in RECEIVE. We couldn't establish more any conclusive outcome though.



  • 6.  RE: MQ Stale connections

    Posted Wed April 05, 2017 12:14 PM

    Morag asked what commands you were issuing.  I believe that she meant 'what MQ commands,' not expecting the need to debug your script. 

    In future posts/replies, may I suggest that issue the MQ commands individually from a shell, then post the MQ command and results from the command.



  • 7.  RE: MQ Stale connections

    Posted Wed April 05, 2017 06:23 PM
    You say "connection counts (CHSTATUS = NONE)". What do you mean by this?

    Sorry, my bad, ts ASTATE(NONE) for the connection.

    Your initial question didn't mention Asynchronous Consumers. Why is this particular state important to you? Are all your connections supposed to be using Asynchronous Consume? ASTATE(NONE) is not a problem if Asynchronous Consume is not in use.

    Observations - Below type are increasing after24 Hrs :-(

    E07DDE5801B4CE22","CHANNELNAME","IP","QMGRNAME","QMGR","NONE","WebSphere MQ Channel","WebSphere MQ Client for Java","MQOO_INQUIRE

    Are the extra QMGR type object handles associated with a channel? Again you can see this in DISPLAY CONN. Are they from a expected source?

    Yes, extra are there as explained above but they are from expected source as per IP verification.

    Look at the CHSTATUS of the additional channel instances (DISPLAY CHSTATUS), are they doing anything? For example, does the number of bytes moved over the channel keep increasing? What is the SUBSTATE? Maybe they are stale connections.

    Yes, bytes are increasing. All Substate are in RECEIVE. We couldn't establish more any conclusive outcome though.

    You have confirmed that the connections are not stale (bytes are flowing over them). Being in SUBSTATE(RECEIVE) means that the  connection is in the perfect situation to be told to do something by the client end. That is the normal resting state of a SVRCONN channel, so that is also fine.

    You say that the connections are coming from an expected IP address, and that all these connections have done is opened the queue manager object OBJTYPE(QMGR) for MQOO_INQUIRE. For a "WebSphere MQ Client for Java" this is one of the first things it does in order to know the queue manager name, platform, version and so on. So for an individual connection this is not out of order.

    Problem is simply that you have "too many" of them. Please now tell us about the configuration at the client end. Are you using connection pooling for example - what is the size of your connection pool? What are these client applications trying to do that might involve making so many connections if you are not using any pooling technology or if the numbers exceed your pool sizes.



  • 8.  RE: MQ Stale connections

    Posted Thu April 06, 2017 12:16 PM

    Thanks for reply.

    Yes, there is pooling for both MDB and Spring frameworks. Here are the config - 

    API Functionality 1 (Implemented in Spring 4.1.2): Total 26 Activity Services
    JMS -> QueueConnectionFactory -> <Connection factory> -> Connection Pool
    Transport – Bindings, then Client
    Connection Pool:
    • Connection timeout : 120
    • Maximum connections : 10
    • Minimum connections : 1
    • Reap time : 180
    • Unused timeout : 600
    • Aged timeout : 0
    JMS -> QueueConnectionFactory -> <Connection factory> -> Session Pools
    Session Pool:
    • Connection timeout : 90
    • Maximum connections : 10
    • Minimum connections : 1
    • Reap time : 150
    • Unused timeout : 480
    • Aged timeout : 0
    Other settings:
    • Polling Interval 5000 Milli Seconds
    • Rescan Interval 5000 Milli Seconds

    API Functionality 2 (Implemented in MDB): Total 5 Activity Services
    Transport – Bindings, then Client
    Connection Pool:
    • Connection timeout : 120
    • Maximum connections : 40
    • Minimum connections : 1
    • Reap time : 180
    • Unused timeout : 600
    • Aged timeout : 0
    Session Pool:
    • Connection timeout : 90
    • Maximum connections : 10
    • Minimum connections : 1
    • Reap time : 150
    • Unused timeout : 480
    • Aged timeout : 0

    Activation Specification
    • Maximum server sessions: 40 

    We are also suspecting and analyzing the connection leakage from CA Introscope in PROD, there seems to be know thread on it - http://www-01.ibm.com/support/docview.wss?uid=swg21605479

    We have opened a PMR with IBM support as well.



  • 9.  RE: MQ Stale connections

    Posted Thu April 06, 2017 10:05 PM

    In Reply to MANISH KUMAR GUPTA:

    We are also suspecting and analyzing the connection leakage from CA Introscope in PROD, there seems to be know thread on it - http://www-01.ibm.com/support/docview.wss?uid=swg21605479

    Does the CA Introscope not have it's own SVRCONN channel? We have not actually seen any of your channel names, so I don't know whether or not you are using the same channel for everything or whether you have various different channel names for each of the different sets of connections. It would be advisable to have some different channels. Then you would be able to tell these applications apart a little better.

    If you had a separate channel for the CA Introscope, and you suspected it of making lots of connections when it shouldn't, you could put limits on the channel using MAXINST so that it's leakage did not affect anything else.

    Read Avoiding run-away numbers of channels

    Cheers
    Morag



  • 10.  RE: MQ Stale connections

    Posted Fri April 07, 2017 11:55 AM

    Hi Morag, thanks for your help.

    Disabling the CA Wily Interoscope has helped a lot so far and now we are waiting for 24 hours to complete to isolate and confirm it as culprit.

    I am discussing your recommendations with team and shall come back if I come across further query.