Hello,

A new distributed MQ application in our organization is facing the challenge of needing to throttle a message consumer due to the back-end system only being able to process 250,000 messages per day. The architect's solution to this is to simply stop consuming messages after the limit is reached, letting the persistent messages build up on the queue for several days in MQ. The initial estimate was that the queue depth could reach 3,000,000 messages over the span of a few days.

Should I be worried about MQ storing a couple million messages for several days? My concerns are with the storage and potentially impacting the performance of the queue manager. I've always read that MQ is not a database and that it's intended for temporary storage of in-transit messages, so I feel responsible for trying to prevent the application from using MQ as part of their throttling solution in order to protect our MQ environment.

Are there any suggested solutions that I can provide to the architecture team aside from MQ?

I appreciate the advice.

Hi Nate,

I hope this reply is of some help to you, though I fear it may not be.

I am left speechless by the statement "The architect's solution to this is to simply stop consuming messages after the limit is reached, letting the persistent messages build up on the queue for several days in MQ."  I really don't understand what hehopes to accomplish by this activity, other than perhaps hopelessly maintain a perpetually growing backlog of unprocessed messages.

There is no intrinsic method for throttling consumer applications in MQ, since one of the objectives of MQ is quick delivery of messages to their consumers.  You should keep after the back-end application team for them to better understand why their system cannot consume moret han 250,000messages per day. 

  250,000 msgs per day == 10,416.66 msgs per hour == 173.66 msgs per minute == 2.89 msgs per second

As to your concern about retaining 3.5 million persistent messages in your queue manager, if your development team and your management all understand and agree that a continuous 24x7 throughput of less than 3 messages per second is acceptable for the back-end, and that MQ is not a throttling point, then you have no concerns - other than to remind everyone that when the input stream exceeds 250k msgs per day -- which is sounds like it does based on your assertion that "the queue depth could reach 3,000,000 messages over the span of a few days", not the 12 days that would be expected at 250,000 msgs per day --  the back-end application will not be able to keep up with the newly arriving messages and at some point

(1) MQ performance may start to suffer as the number of messages in the queue begins to exceed only a few multiples of 10,000 with active simultaneous MQGET and MQPUT operations

(2) MQ storage utilization will continually increase; if your messages average 4K in size, your may need as much as 30GB of storage to hold all these messages.

Good luck.  lertr us know what happens.

Dave

 

This is a classic fast producer/slow consumer problem and I sometimes go beyond Dave's recommendations in solving it.  If I reduce the queue depth, disallow the application or channel access to the DLQ, and make similar restrictions all the way back up the pipeline to the producer, then instead of MQ clogging up, the producer eventually gets a QFULL error and throttles itself.  As the consumer processes messages, the backlog frees up queue space.  That cascades through the pipeline until space frees up at the producer end and it can put more messages. This way the producer is bound by the speed of the consumer and MQ isn't responsible for housing multiple days worth of messages.

Also, would it be a problem if you lost the queue holding those 3M+ messages?  If they can be recreated easily then you really don't need to manage 3M messages in MQ.  Just discard the overflow and let the apps at either end reconcile the discrepancies.  But if it would be a large impact to lose those messages, then are the logs linear or circular?  Because you don't get media recovery with circular logs and losing a queue means losing all messages on it.  But if you enable linear logging to account for the unusual storage requirement the entire QMgr takes a ~20% performance hit right off the top.

I'm willing to let MQ absorb transient peaks to smooth and dampen the message volume curve.  But what you describe goes way beyond that.  Ultimately, message producers are limited by the capacity of the message consumer.  When you get past the normal traffic-shaping effect that a queue provides MQ will not overcome a basic capacity shortfall at the consumer.  When you begin to queue up excessive volumes of messages you are also accumulating risk.  It is imperative to consider at what point that risk becomes excessive and address the capacity issues before then.  At 3M+ messages it's probable you passed that risk break-even point a long, long time ago.

IMO there's a fundamental design issue here, not an MQ problem. The solution is not for the target app to stop processing either. Many questions arise about the scenario - Does the target want all the messages it's getting? Is there middleware inbetween that could filter some way? Why is the application so slow in processing? Could it store messages to a temporary table to vet them before processing?

It's also not up to the producing application to curb its message output either, the downstreams need to deal with the traffic. Assuming of course that the producer is not issuing unnecessary messages.

These are the things the architecture team needs to consider.

Regards, Rick

I was able to raise my concerns to higher-level architects and they eventually agreed that the application's design to use MQ as the throttling point was not a good idea. The architecture team decided to follow up with the reasoning for needing to throttle the consumer in the first place and found out that the 250k throughput was an arbitrary number that someone made up without doing the research.

Thank you for your help, everyone!