MQ

Expand all | Collapse all

runmqsc and amqpcsea error

Jump to Best Answer

  • 1.  runmqsc and amqpcsea error

    Posted 27 days ago
    Edited by Bernard Pittens 27 days ago
      |   view attached
    Hi,

    We have monitoring on our MQ Appliances established via  a script like below and we use a command to execute it remotely on a Windows 2012 R2 server since 2017 without a problem. But now we see some strange behavior,  (We use MQ Appliance M2000B 8.0.0.13 )

    runmqsc -w 120 -c QMAN_PRD  <  script.mqsc  >  outputGensyssim.txt

    Content script.mqsc:
    --------------------------
    *Display QMGR
    dis qmgr VERSION
    *End Display QMGR

    *Channels autoscan
    dis chl(*)
    *End Channels autoscan

    *Channel status
    dis chs(*) STATUS BYTSSENT BYTSRCVD MCASTAT
    *End Channel Status

    *Local Queue Monitor
    display ql(*) descr process crtime crdate ipprocs opprocs curdepth
    *End Local Queue Monitor

    *Display PUBSUB
    dis pubsub status
    *End Display PUBSUB
    -------------------------------------

    Since this week most times when this script runs it puts a message on the DLQ of the Qmanager with these properties:
    Put application name: amqpcsea
    Destination queue: AMQ.5F728DF62BF23AFF
    Reason: MQRC_Q_FULL
    (see also attachment)

    We see that when it takes more time to run the script at the beginnen of the proces it puts the message on the DLQ.  If the script runs quickly this strange message is not send to  the DLQ.
    On the Qmanager logs we don't see any error, also in lower environment we don't have this problem.
    We use about 300 queues, 40 topics with subscriptions and 10 channels.

    Any idea what the problem is and how to solve this ?

    Thanks.

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------


  • 2.  RE: runmqsc and amqpcsea error

    Posted 27 days ago
    Edited by Bernard Pittens 27 days ago
    Hi,

    I think we found the problem, when we remove the Channelstatus Check lines in the script.mqsc :

    *Channel status
    dis chs(*) STATUS BYTSSENT BYTSRCVD MCASTAT
    *End Channel Status

    No problem with the DLQ occurs, and we also see no error in the output (outputGensyssim.txt) of the call. Apparently the status channel request write 12000 lines to the logfile.  
    We have 10 channels and one channel is writing about 12000 lines of logdata  (this is a channel for traffic from legacy AS400 systems).

    What for me interesting to know is why this is happening ?
    And can I define a backout or error channel for this so that the runmqsc output is not send to the DLQ but to a self defined extra .err queue ?
    Kind regards



    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------

    Original Message


  • 3.  RE: runmqsc and amqpcsea error
    Best Answer

    Posted 27 days ago

    Each answer from the MQ Command Server is one message. So for example, to display 300 queues, the ReplyQ that the runmqsc tool uses (AMQ.5F728DF62BF23AFF in your example) would have 300 messages placed on it as a result of the command displaying all the queues.

    If you have a lot of running (or stopped) channel instances (you suggest 12000 in your second post), in answer to a DISPLAY CHSTATUS command, then that would result in a lot of response messages.

    The reply queue used by runmqsc is created from the model queue SYSTEM.MQSC.REPLY.QUEUE which, by default has a MAXDEPTH of only 3000. It would be perfectly appropriate to increase the MAXDEPTH of the model to a number big enough to allow the command server to place all the response messages onto the runmqsc ReplyQ. I suggest that this would be the simplest way to get all the responses back from the command server, and thus understand why you have so many channel instances.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------

    Original Message


  • 4.  RE: runmqsc and amqpcsea error

    Posted 27 days ago
    Hi Morag,

    Thanks very much for your response, yes increasing the maxDepth solves the problem, I also received a link via Josh McIver(thanks for that) and he pointed to this link: 

    https://t-rob.net/2018/05/03/client-based-runmqsc-gotcha/

    What T.Rob is writing is true in my opinion, this is an IBM problem because as an engineer you are not warned when the response of runmqsc is not OK:

    Kind Regards

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------

    Original Message


  • 5.  RE: runmqsc and amqpcsea error

    Posted 26 days ago
    Back in 2018, I did some testing with regard to T.Rob's blog posting and posted the results on my blog. 

    Regards,
    Roger Lacroix
    Capitalware Inc.

    ------------------------------
    Roger Lacroix
    ------------------------------

    Original Message


  • 6.  RE: runmqsc and amqpcsea error

    Posted 26 days ago
    Edited by Tim Zielke 26 days ago
    I agree wholeheartedly this is a bug, and an insidious one at that.  One of the last things you want from runmqsc is missing data with no indication (especially from the client side) that something went wrong.  I believe runmqsc in the client mode is sending PCF commands to the command server to get the data back for the sent command (e.g. DIS CHL).  I have a similar C client program that we use extensively to run PCF commands in this fashion in our shop.  When T.Rob noted the community of this bug, one thing I did was to update my code to count the number of PCF replies that come back for a given PCF request and then check if the msgSeqNumber from the last PCF reply message matched the total number of replies.  If they do not match, the program prints out an error to the user that reply messages were missing.  Although this approach is not bulletproof, it at least catches most typical cases when replies would be lost due to the queue full condition on the reply queue.  To properly fix this, IBM needs to do something like enhance PCF to send back both the msgSeqNumber and a TotalReplies value in the MQCFH header so a program can properly detect when a PCF reply was not sent back to the client.  This is getting into data integrity and needs to get fixed.

    <Edited>
    I just remembered that the last PCF reply message sets the Control field in its MQCFH header to MQCFC_LAST.  So I would think as long as the runmqsc client counts how many reply messages it receives and also makes sure the msgSeqNumber on the last reply (i.e.  MQCFH.Control=MQCFC_LAST) matches the count of reply messages, the runmqsc client could accurately detect if any reply messages were missing and report this to the user.

    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 7.  RE: runmqsc and amqpcsea error

    Posted 26 days ago
    Edited by Francois Brandelik 26 days ago
    Hi Bernard,

    Look at your model queue that you are using. You need to give that model queue a greater max q depth so that when the command server replies you have enough space to receive all the replies. Remember the reply queue is created off the model queue and will take it's max qdepth attributed. If you get MQSC replies onto the DLQ with Q_FULL obviously you need a greater max depth on the model queue...  (SYSTEM.MQSC.REPLY.QUEUE)

    Obviously this happens only in larger installations but is to be considered a bug. The work around is clear.
    Hope it helps

    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message


  • 8.  RE: runmqsc and amqpcsea error

    Posted 26 days ago
    FYI, I submitted a PMR for IBM to confirm if the runmqsc client has the proper integrity check to confirm if the expected reply messages were actually received by the runmqsc client, and if not report an error to the user.  There is no acceptable workaround here.  This is a critical data integrity requirement that the runmqsc client accurately detects missing reply messages and reports that back to the user in some form of error.

    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 9.  RE: runmqsc and amqpcsea error

    Posted 9 days ago
    In the PMR, IBM confirmed that runmqsc has the proper integrity check to report an error if the msgSeqNumer on the last reply message does not match the total reply messages received by the runmqsc client. I did some of my own testing by lowering the maxdepth of the SYSTEM.MQSC.REPLY.QUEUE to low values (e.g. 1, 100, 500, etc.) and indeed you do eventually get an error like "AMQ8416I: MQSC timed out waiting for a response from the command server to queue 'AMQ.5FB5DD31231CE411'." reported back to the runmqsc client after it times out on the reply queue. So if scripting is not properly checking the AMQ results that come back and flagging AMQ messages that are non-standard, that would be an issue on the scripting side.

    I would also note that it is somewhat misleading how the interactive runmqsc client will return a partial list of results and then hang for the timeout on the reply queue (which looks like it is 30 seconds) but also return control back to the user. So for example, you could run the command DIS QL(*), get back a partial list and control to enter new commands, enter a second command like DIS CHL(*), and then when the reply queue times out you get the AMQ8416I error immediately followed by all of the DIS CHL(*) results. So it would be easy to miss the AMQ8416I error in the flurry of return data and think the DIS QL(*) was successful. I think it would be better if the runmqsc client returned no result data until it confirmed it had all the reply messages, and if not, just display an error with no result data returned.

    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 10.  RE: runmqsc and amqpcsea error

    Posted 7 days ago
    Edited by Tim Zielke 7 days ago
    One other note here. I did some more testing, and it looks like runmqsc on Linux does set the return code correctly when it encounters issues. 

    Here is an example of a good run where runmqsc runs successfully and the subsequent "echo $?" has a 0 return code.

    > echo "dis chl(*)" | runmqsc -c QM1
    5724-H72 (C) Copyright IBM Corp. 1994, 2020.
    Starting MQSC for queue manager QM1.

    1 : dis chl(*)
    AMQ8414I: Display Channel details.
    CHANNEL(CHL.TO.QM2) CHLTYPE(SDR)
    AMQ8414I: Display Channel details.
    CHANNEL(CHL.TO.QM3) CHLTYPE(SDR)
    .
    .
    AMQ8414I: Display Channel details.
    CHANNEL(CHL.TO.QM100) CHLTYPE(SDR)

    2 command responses received.

    > echo $?
    0


    Here is an example of a bad run where runmqsc times out on the reply queue and the subsequent "echo $?" has a non-zero return code.

    > echo "dis chl(*)" | runmqsc -c QM1
    5724-H72 (C) Copyright IBM Corp. 1994, 2020.
    Starting MQSC for queue manager QM1.


    1 : dis chl(*)
    AMQ8414I: Display Channel details.
    CHANNEL(CHL.TO.QM1) CHLTYPE(SDR)
    AMQ8414I: Display Channel details.
    CHANNEL(CHL.TO.QM2) CHLTYPE(SDR)
    AMQ8416I: MQSC timed out waiting for a response from the command server to queue 'AMQ.5FB5DD3128766C05'.
    0 command responses received.

    > echo $?
    10


    So it looks like besides validating AMQ messages that are returned from a runmqsc client call within Unix scripting, another scripting validation is to check the return code of the runmqsc command and validate if it is zero (successful) or not. 

    I have also updated the mqsc-qmgrs script in the following blog post to give an example of how to check the return code for a runmqsc client invocation.

    https://community.ibm.com/community/user/middleware/blogs/tim-zielke1/2020/07/09/mqsc-qmgrs-script
    ​

    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 11.  RE: runmqsc and amqpcsea error

    Posted 5 days ago
    Hi Tim,

    Time wrote:
    >I think it would be better if the runmqsc client returned no result data until it confirmed it had all the reply messages,
    >and if not, just display an error with no result data returned.

    This means that runmqsc would need to store the intermediate result data, and then display the data only if all reply messages were received (with the timeout). I imagine this would require significant code changes in runmqsc, given that the number of MQ objects returned could be essentially unlimited (tens of thousands is not uncommon).

    I believe dmpmqcfg has a similiar issue.

    Cheers, Glenn.

    ------------------------------
    Glenn Baddeley
    Senior Middleware Software Engineer
    Coles Supermarkets Australia Pty Ltd
    ------------------------------

    Original Message


  • 12.  RE: runmqsc and amqpcsea error

    Posted 5 days ago
    Edited by Tim Zielke 5 days ago
    Hi Glenn,

    Yes, I agree with you that runmqsc would need to store the results in memory before printing them out with this change.  However, with the amount of memory that is available in a 64 bit virtual address space, I would think that would not be a problem. :-).  As far as I know, any supported version of distributed MQ is now 64 bit.  Also, probably even a 32 bit address space would suffice for this requirement, too.

    Thanks,
    Tim

    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message