MQ

 View Only
  • 1.  Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Tue May 10, 2022 05:51 AM

    I am in a very small development team that has no experience of Websphere and we have inherited a legacy system that communicates with a third party system via Websphere queues.

    Most of the time everything ticks along happily, however we are beginning to get times when traffic increases to nearly 5,000 messages in the space of a few seconds and our poor old system is taking almost 15 minutes to put them onto a queue.

    I don't expect anyone to tell what is wrong from this basic description, I am simply wondering if I am right in thinking putting 5,000 messages to a queue could or even should take a lot less time? Like maybe a few seconds to a minute tops sort of thing.

     

    More details for those that are interested:

    By adding a load of logging to a version we put on a test server we were able to rule out the other activities of the code base (such as writing to databases and so on).  We were seeing about 150 milliseconds per "put" command.

    Our Websphere queues are hosted on servers in Brazil with our system hosted with AWS in East US but the administrators in Brazil assure us that this difference in location is not an issue.

    The C# code sends the messages in batches of 200 by starting a transaction on the queue, looping through each message and putting them to the queue, updating a database for all 200 messages and then committing the transaction (the DB update is pretty much instant).

    This being legacy code the initial temptation is to assume they have done something horrifically wrong but, being a websphere newbie myself, it is going to be tricky for me to spot it!



    ------------------------------
    Philip Bathe
    ------------------------------


  • 2.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    IBM Champion
    Posted Tue May 10, 2022 06:11 AM

    You are correct in thinking that 5000 messages in 15 minutes is slow. My little laptop here can put 5000 persistent messages (10000 bytes in size) in around 17 seconds - that's seconds not minutes.

    What version of MQ are you using? I assume an old-ish one as you are still using the WebSphere brand to refer to it. There's no point in us advising you things to try if they are not available on your version of MQ.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------



  • 3.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    IBM Champion
    Posted Tue May 10, 2022 06:28 AM

    As Morag says I would expect it to be in seconds, not in minutes in a well set up environment, where the application is close to the server.

    I presume you are using a client connection, and you are measuring the duration the mqput takes, not the duration of a "MQPUT and wait for a response message"

    Some questions which may help pin it down
    1 Are you using persistent or non persistent messages?   non persistent have no logging
    2 Are you using in syncpoint or out of syncpoint - see below
    3 What size are your messages.   Big messages take longer than small messages - anything under 10KB is small.  Anything over 100KB gets to be large.
    4 If your client is distant from your server, you may get network delays (especially if your messages are large).  How long does a ping take?  under 10 ms is good.

    If you want to "play"  to see the impact
    1 make the messages non persistent
    2 use a small message (100 byte)  and measure the time for the put.


    Syncpoint
    There are two modes of putting messages
    Put in syncpoint, put in syncpoint,put in syncpoint, commit ( which writes to the log, and the messages are visible)
    Put out of syncpoint (which does put, commit under the covers) Put out of syncpoint (which does put, commit under the covers)...  (no commit, if there is a commit, it has nothing to do)

    so
    1 puts in syncpoint should be fast, and the commit - will be slower because of the log write.
    2 put out of syncpoint will be slower than put in syncpont , because they have to do log I/O.

    If you are doing database updates as well, I hope your requests are in syncpoint, so your commit does both database and MQ .

    Colin





    ------------------------------
    Colin Paice
    ------------------------------



  • 4.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Tue May 10, 2022 10:01 AM
    Thanks for the clues.

    We do capture the message ID after we put them on the queue but if you mean do I include the wait for a response from the 3rd party app (which comes on a different "OUT" queue) then no I am only measuring the time for the MQDestination.Put method to complete.

    Answering your numbered questions:

    1 Are you using persistent or non persistent messages?   they are persistent
    2 Are you using in syncpoint or out of syncpoint - we do use syncpoint
    3 What size are your messages - I am only logging the message length which is roughly 340 characters so it looks like these are nice and small (I am not sure how to get the message size proper yet)
    4 If your client is distant from your server, you may get network delays (especially if your messages are large).  How long does a ping take?  Is there a special technique to pinging MQ queue servers - simply pinging the host domain resolves to the AWS server and times out

    I shall try turning off persistence next and see what that does.

    ------------------------------
    Philip Bathe
    ------------------------------



  • 5.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Tue May 10, 2022 10:04 AM
    Edited by Philip Bathe Tue May 10, 2022 10:11 AM
    Sorry I think my reply got lost somewhere.

    The queues are on 9.2.5 but the code is using version 7.5.0 of amqmdnet.dll.

    "Websphere" is our shorthand for the special IBM queues as opposed to the MSMQ queues we also use.

    ------------------------------
    Philip Bathe
    ------------------------------



  • 6.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Tue May 10, 2022 11:07 AM
    Then I am not tilting at windmills here, thanks :-)

    The queues seem to be on 9.2.5 (I am going by the MQ Explorer > Queue Manager XXX > Properties > QuickView > Version being 09020500)

    The code looks to be using version 7.5.0 of amqmdnet.  I think trying a more recent version of this will not hurt any.

    My Websphere talk is because this is how we describe these special queues within my team (as opposed to the more common MSMQ which are used by a number of the smaller apps we maintain) and no one has known any different until I started digging this week.

    ------------------------------
    Philip Bathe
    ------------------------------



  • 7.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Tue May 10, 2022 11:44 AM
    Hi Philip

    From a quick web search, a round-trip latency of 125ms doesn't seem to be unusual for AWS-US-East to AWS-Sao-Paolo (you don't say whether the Brazil side is on AWS or not, but in any case that seems like a good baseline).  Despite your admin's assurances, this is a bit of a stretch for the MQ client (not unsupported, but with potential problems as you have seen).

    Unless I'm missing anything, as you've described the application design, you will be waiting for a 'line turnaround' on every message sent - the next message cannot go on the wire until the previous has been acknowledged.  With that kind of latency involved, it would explain the slow behaviour you are seeing.

    There are a few things you could do to alleviate the problem.  The most obvious starting from a 'green field' would be to have a queue manager local to the sending application take the messages and 'buffer' them - this kind of store and forward processing is bread and butter to IBM MQ, and you would hand off worrying about the latency between the locations entirely to MQ (which can be smarter about batching the messages over the wire etc. anyway when talking Queue Manager to Queue Manager).

    Another option is to investigate enabling 'Async Put' from the MQ client - which allows the client to stream multiple messages without waiting for acknowledgment.  However, I think this would need application code changes to periodically check that batches have arrived using MQSTAT to have the same confidence in message delivery that the current design gives.

    Regards
    Anthony

    ------------------------------
    Anthony Beardsmore
    IBM MQ Development
    IBM
    ------------------------------



  • 8.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Tue May 10, 2022 12:20 PM
    Thank you, your words have reinforced what I was beginning to suspect from Morag's and Colin's help earlier.  I had also googled AWS latencies and I have just started asking to get our test queues moved to East US to prove the theory. 

    Yes the queues are also on AWS and the bonus part is that the 3rd party system (that sits the other side of these queues) is also in East US - so a double latency whammy! I believe we have a number of systems based in Brazil and probably having all the MQ queues in one place was just easier.  Anyway you don't need to hear the history...

    Thanks again to everyone for their time.

    ------------------------------
    Philip Bathe
    ------------------------------



  • 9.  RE: Am I right in thinking 15 minutes to put 5,000 messages to a queue is very slow?

    Posted Wed May 11, 2022 09:24 AM
    On the client side if you have access to root or know an admin, maybe doing a tcpdump may shed light on the subject. Capture traffic using the dest host and port.  You can then use something like WireShark to look at the packets.  Should be able to see the MQ PUT, MQ CMIT and ACK packets.

    We had an issue where messages were being read slowly from a client and they blamed us.  Come to find out they did the GET then waited until the message was written to their database which was slow before sending us the COMMIT. The delta from the GET to the COMMIT was seconds!

    ------------------------------
    Earle Ake
    ------------------------------