I've discovered an issue in runmqsc when running in client mode that causes it to return partial result sets and when it fails it does so silently.  I have an extensive write-up on my blog and rather than reproduce it here I'll just provide a link. 

The "Executive Summary" version is that Command Server results are unreliable when any PCF command issued from a remote host via MQ client results in more than 3,000 responses, and it becomes more unreliable as the number of responses increases above that number. 

I first encountered this when issuing DIS QCLUSTER commands from runmqsc in client mode with a result set of just over 4,000 objects.  The results were incomplete about 60% of the time and there were no error messages on the QMgr side to indicate what went wrong.

The blog post has some defensive programming suggestions but the quick-and-dirty solution is to increase MAXDEPTH of SYSTEM.MQSC.REPLY.QUEUE and any other model queues used by the Command Server to a very high value.  That doesn't address the underlying problems with the exception handling in the Command Server and runmqsc but I've got a PMR open for that.

T.Rob,

This is yet another example of a MQ user/administrator being hurt by a maximum Queue Depth setting.  In this particular case, this issue hit one of the more experienced MQ administrators in the world and it still took considerable time to resolve.  Resolving this issue was made doubly difficult because the software component experiencing the problem wasn't able to directly communicate the issue back to the user experiencing the problem.  This is actually rather typical in modern environments and will become even more commonplace in the coming multi-cloud environments.  

The point here is that, with proper Queue Manager configuration, this problem never needed to be encountered!  See the link below to a thread (on this forum) discussing the danger of the Maximum Queue Depth setting.  

 Queue Depth: Are you at risk?

Regards,

Glen Brumbaugh

Hi Glen,

Thanks for the read and comments.  I saw the other thread and stayed away because I had recently rebutted a couple of your posts and didn't want to appear to be piling on or to bear a grudge. (Most people don't know there's a photo of you and I in my kitchen and that we love to argue these kinds of things.)  Also, in the case of the certificate info I can point to an authoritative standard whereas this one is more opinion based so I didn't see it as urgent. That said, I do not at all believe that this incident supports an assertion that MAXDEPTH is in itself dangerous and I can't leave a comment stating as much hanging unanswered in the thread.

For the record, my position is that MAXDEPTH is a useful control and I resist the tendency to arbitrarily set it to values that render it useless for its purpose.  From the perspective of a security engineer it is not merely useful but one might say mandatory.

> This is yet another example of a MQ user/administrator being hurt by a maximum Queue Depth setting.

I don't know the reasoning behind IBM's decision to set MAXDEPTH to 3,000 so I can't be sure but I tend to agree with you on this one.  Every large shop that I have ever worked for or have knowledge of would bump against a MAXDEPTH limit of 3,000 on this queue.  Reducing  that value from the 5,000 used almost everywhere else seems ill considered.  Realistically the default should either represent the top percentile of possible result sets or be unlimited.  (In the latter case I'd still set it lower, just like I routinely do with the DLQ, but I'd use a value like 50k as default.)

> Resolving this issue was made doubly difficult because the software component experiencing the problem wasn't able to directly communicate the issue back to the user experiencing the problem.

True but this can be addressed in the design.  The root cause as I see it here is one of poor exception handling.  The runmqsc client was issuing GET with extremely low wait intervals, then stumbling when the queue emptied before the last message was received.  It gives an error stating there are responses on the queue from a previous command when we know - because it's a dynamic queue - that these are from *this* *instance's* own prior command and it doesn't recognize that.  MFT has all these same issues but is able to resume a transfer because the two sides talk to one another to manage state.  Here, runmqsc and amqpcsea are more like estranged family at the reunion - they reluctantly talk to one another as familial duty requires, but as soon as one of them is slightly insulted it all goes to hell.

But even if we set aside the issue of a fragile protocol, the software component experiencing the problem does have the ability to notify the MQ Admin but does not.  Maybe we don't want to make amqpcsea as robust as an MFT agent but it doesn't seem like an error message on QFULL is a lot to ask given that this condition currently breaks the function being provided.  For a couple decades now we've been able to point to IBM components as the Gold Standard for coding best practices because they do things like check the Message Type to make sure they aren't processing report messages.  This falls far short of that standard.

> The point here is that, with proper Queue Manager configuration, this problem never needed to be encountered! 

Here I definitely disagree because 'proper' can only be defined in relation to specific business requirements and there are legitimate requirements that conflict with regard to MAXDEPTH tuning.  But let's assume I'm willing to engage in defensive tuning and impost a categorical standard, the IBM developers aren't the ones I'm supposed to have to defend against. If I'm using IBM-written administrative components I should get back either a valid result or a useful error. 

The root causes here are poor exception handling and failure to provide diagnostic information.  A system in which a critical administrative component is easily broken and doesn't raise an alert when it fails isn't the kind of thing you want to fix by tuning around the problem and leaving the vulnerability in place.  A better description of such a system is that it's a CVE just waiting to happen.  Setting MAXDEPTH up makes a proper fix less urgent but no less important, in my opinion. 

T.Rob,

I always enjoy our discussions and I like to think that they help to illuminate some of the intricacies of MQ for others.  The link that you provide in your post is to your excellent description of the issue with the runmqsc command.  I think that we both believe that the Maximum Queue Depth setting for the queue in question is unfortunate. I would expect to see from the Lab some combination of modifying the Max Queue Depth, message persistence, and message expiry for the processes involved. There are multiple tuning options to resolve this issue.  Personally, I would be more comfortable with a much higher queue depth and a relatively short expiry period (e.g 5 minutes). 

It's hard for developers to get these kinds of settings right the first time.  The neither have the laboratory facilities to test large volumes nor the operational experience to have those lessons ground in deep.  Unless you've worked in an Enterprise with 1,000s of Queue Managers you simply haven't had the hands-on opportunity to learn those lessons.  If the developers have perfect foresight, they can anticipate these issues.  Unfortunately, they don't have the facilities to test them even if they do anticipate them.  That's why we have Alpha and Beta releases.  

Now, back to the point of my post.  Any configuration which is the MQ administrator's responsibility and which has to potential to break a production environment is "dangerous".  You know me, I like to be provocative to get people thinking.  It's not that the setting itself is dangerous, it's that not respecting the setting,  or not being aware of the setting, or not managing the environment so that the setting doesn't take down production are dangerous.  It's also dangerous to not pay attention to x.509 certificate expiry dates.  Those two things will both bring down a Production environment.  Actually, in my experience, certificate expiry and maximum queue depth are probably the two most common causes of preventable production outages.  

A final note on this particular queue attribute.  It has be seen and discussed as either being useful for disk space management or for queue monitoring.  From a storage system point of view, on distributed MQ platforms there is no difference between a file and a queue.  How many UNIX or Windows OS versions manage file space usage by limiting the number of records?  One terrabyte length record would run most files and queues out of space.  

How about Queue monitoring.  Min and Max depth monitoring has been problematic since day one of MQ.  For a long time, despite its being a crude tool, it was all we had for monitoring.  We now have much better statistics to use to determine if applications are reading or writing messages from/to a queue.  This attribute is a hold-over from the days when MQ was a young product.  As engineers, administrators need to be aware of both the potential benefits as well as the potential costs of these parameters.  My post was a call to pay Attention!   

In the end, you and I probably see 95% eye to eye on almost all of the MQ issues that we discuss and just differ in either personal style or in the implementation trade-offs that we choose to accept.  In the case of this attribute, I do the following:

• Set all queues to have max depth.  This includes the System queues.  This is built into my "Golden Queue Manager" script and run against all Queue Managers.

• Set monitoring alerts for all queue depths over 5,000 messages.  These alert values are initial values for the alerts and, like all alerts, need ongoing tuning.  

• Monitor system disk space for both queues and logs.

There are, of course, other ways to achieve the same results.  I always try to separate out style from substance.  It's never about doing things "Glen's" way.  It's about keeping Production as stable, resilient, and reliable as possible.  Multiple approaches can get an environment to the same place.  Different shops have different constraints based upon personnel, tools, support, etc.   I just want, as do you, administrators to know their choices and to make wise decisions.  

I don't feel that I have anything further to say regarding Maximum Queue Depth, but if I have missed something please feel free to add a final comment.  

As always my esteemed colleague, your friend,

Glen