MQ

QMgr v7501 on multiple Win2008R2  servers- ends mysteriously, everyday at same time, and then off and on, through the day, weekend.  I cannot says for sure, but the QMgrs seem to be acting like the OS is running out of resources.

The MQ error logs show nothing; the QMgr logs just show 'qmgr is ending'. I think the Windows OS is deciding to shutdown services due to lack of resources: cpu, memory, disk.    My non-Windows person opinion.

 

I am the MQ administrator but not the server owner, so my exposure is limited.

A PMR to IBM was opened , but IBM wants QMgr patched up to v7508 first.  Server owner wants me to keep researching for they do this.

 

Just wondering if anyone else has input into where to look to discover why the BM MQ Series service is stopping. From the Event Log viewer, I do see that the OS does try to start the QMgr again, about 30 seconds after issuing 'endmqm' command, but since there are other applications that use MQ, the QMgr cannot restart until the applications are done or  'let go' of the previous processes(QMgr). 

 

I will accept all input as I have hit a wall now.

Any error logs ?

On Wed, Feb 28, 2018 at 9:05 PM, marge walker <wsmqfam-ws@lists.imwuc.org>
wrote:

> QMgr v7501 on multiple Win2008R2 - ends mysteriously everyday at same time
> -----End Original Message-----
>

either in /var/mqm/errors or /var/mqm/qmgr/errors

On Wed, Feb 28, 2018 at 9:13 PM, Vinay kumar <wsmqfam-ws@lists.imwuc.org>
wrote:

> Any error logs ?
>
> On Wed, Feb 28, 2018 at 9:05 PM, marge walker <wsmqfam-ws@lists.imwuc.org>
> wrote:
>
>> QMgr v7501 on multiple Win2008R2 - ends mysteriously everyday at same time
>> -----End Original Message-----
>

Vinay - both the QMgr and the MQ error logs show nothing - not even an FDC!.  The pmr I opened with IBM - the IBM person stated the same thing.  He wants the QMgr patched up before he dives in again.  Quite frustrating as I have been an MQ admin for 8 years now.  On all OS platforms. 

That's a common answer we can expect from anyone if we say there is no logs
to verify the issue , most importantly when you say issue is happening at
the same time every day we can assume few things but again they are just
assumptions which can't be used to prove things until we have a proper logs
or FDC to verify things.

1) Was there any sudden spike in the number of connections to the qmgr ?

2) Was there any load on the qmgr in both ways either connection wise or
either disk wise or memory wise .

3) Just verify ps -ef |grep qmgr just before qmgr goes down as it was
mentioned this is happening every day ?

4 ) run runmqras before qmgr going down and after qmgr coming up as well .

As always keep qmgr in latest version as presently version 9 is also
available better to move to atleast 8 version to rule out some unknown
issues .

Regards


On Wed, Feb 28, 2018 at 9:33 PM, marge walker <wsmqfam-ws@lists.imwuc.org>
wrote:

> Vinay - both the QMgr and the MQ error logs show nothing - not even an
> FDC!. The pmr I opened with IBM - the IBM person stated the same thing.
> He wants the QMgr patched up before he dives in again. Quite frustrating
> as I have been an MQ admin for 8 years now. On all OS platforms.
>
> -----End Original Message-----
>

moreover just have a keen look at the qmgr going down logs to get some
sight of the issue ,i know you may already checked all the things as per
your experience but just giving details from my side .

On Wed, Feb 28, 2018 at 10:05 PM, Vinay kumar <vinaykumar547@gmail.com>
wrote:

> That's a common answer we can expect from anyone if we say there is no
> logs to verify the issue , most importantly when you say issue is happening
> at the same time every day we can assume few things but again they are just
> assumptions which can't be used to prove things until we have a proper logs
> or FDC to verify things.
>
> 1) Was there any sudden spike in the number of connections to the qmgr ?
>
> 2) Was there any load on the qmgr in both ways either connection wise or
> either disk wise or memory wise .
>
> 3) Just verify ps -ef |grep qmgr just before qmgr goes down as it was
> mentioned this is happening every day ?
>
> 4 ) run runmqras before qmgr going down and after qmgr coming up as well .
>
> As always keep qmgr in latest version as presently version 9 is also
> available better to move to atleast 8 version to rule out some unknown
> issues .
>
> Regards
>
>
> On Wed, Feb 28, 2018 at 9:33 PM, marge walker <wsmqfam-ws@lists.imwuc.org>
> wrote:
>
>> Vinay - both the QMgr and the MQ error logs show nothing - not even an
>> FDC!. The pmr I opened with IBM - the IBM person stated the same thing.
>> He wants the QMgr patched up before he dives in again. Quite frustrating
>> as I have been an MQ admin for 8 years now. On all OS platforms.
>>
>> -----End Original Message-----
>>
>
>

Anything in the widows event logs (Event Viewer)?

Rab,

Yes, I scanned Application Logs, System Logs, Security Logs.  The server owner and I found very little to target.

Is these servers running on VMs or physical servers ?

Thanks
Mahesh

On Fri, Mar 2, 2018 at 11:23 AM marge walker <wsmqfam-ws@lists.imwuc.org>
wrote:

> Rab,
>
> Yes, I scanned Application Logs, System Logs, Security Logs. The server
> owner and I found very little to target.
>
> -----End Original Message-----
>

First - I got this issue resolved, finally. It was a password issue on the Active Directory service account that runs the IBM MQ service on this Windows server.

I tracked a different AD service account, that was logging in, and immediately after it did, the QMgr would issue 'endmqm.exe' . No tracing, no logging, just the special AD service account logging in and then the immediate shutdown of the QMgr. I tracked down the owner of the powerful service account and battled with that person for 1 week. They finally researched how the MQ service account was setup in the EPAR (password repository). Appears that when someone changed the MQ service account password in mid-January, they marked some field incorrectly, hence allowing/demanding that the special service account log into this service multiple times a day and disable the password for the MQ service account. Lovely, huh? And all along, the owners of this Windows server where receiving emails from this EPAR group, telling them there was an issue with the password. Yet, they failed to inform me, the MQ administrator, about it. ahh - the gloriousness of the Black Hole.

 

Second - thanks to all who helped with with your MQ advice!! It was kind of fun.

Congratulations.  (It always amazes me how people just ignore the alerts configured to help them.)

The same circumstances occurred at a project I worked at some time ago. The queue manager (not IBM MQ however) went down at exactly the same point in time every night for over a week. Again - no logs and it started one night, apparently for no reason.  I am not saying that this will solve your problems, since I do not know your exact circumstances, but the symptoms seem to indicate something similar.

>> https://developer.ibm.com/answers/questions/210867/virus-scanner-on-my-system-with-mq.html

>> "MQ requires exclusive locks over its transaction log files. If a Queue Manger’s logs are locked by another program and the MQ processes cannot access them, the Queue Manger will terminate in order to preserve log integrity. This also applies to the MQ data files."

If it is not a virus scanner - and you are using Virtual Machine Snapshot backups - I would definitely take a look at this as well! Again it will try to allocate MQ files - even though it might only be for read purposes - an MQ will not like it.

The queue manager not being able to shut down and applications hanging around is quite common and it is the application's fault. Tell the application programmers they HAVE to use the option FAIL_IF_QUIESCING in all of their API calls. This should return a failure to the application when the queue manager is trying to shut down and thus cut the connection allowing the queue manager to shut down gracefully.

 

> ... everyday at same time, and then off and on, through the day, weekend...

Can you say more about the "then off and on" part... only weekends?  only daytime?

What are the windows for administrative operations (backup, archiving, network maint)?

I also think both Rab and Johan have provided excellent ideas.  A/V is often a periodic problem with systems intended to be continuous.

Thanks Jeff - and to everyone that took the time to give me input.  I do believe it is a system resource-shortage that is causing this.  Something is getting pushed to these servers, starting at the same time, daily, which I believe is maxing out some system resource (memory, paging?)  On a few machines, the  QMgr tries to restart itself (strmqm.exe) but fails due to application not letting go of previous occurrence.  Messy!  I am working with Performance group now - to run reports for me on a few of these servers.  And, it would be great if the application folks would fess up if they made a change to their application recently.   right?  As soon as we get this resolved, I will post.  At this point, I am also engaging HPOO to catch the  'QMgr is stopping:  get into server, stop applications, stop QMgr, start QMgr, start application' - a Band-Aid at best.

Marge,

You mentioned you saw 'endmqm' issued in the event logs, who/what issued it? 
Is access to the mqm user ID and group (and MQ admin commands) locked down?

Recurring behavior sounds like a scheduler task behavior.

Tracy - the IBM MQseries service is run by a ActiveDirectory service account.(also a member of domain mqm)   In the Win event logs, this account is issuing the endmqm, dspmq, and strmqm commands. Then it goes silent until human intervention.