MQ

 View Only

 DLQ monitoring using activity trace?

John Hawkins's profile image
John Hawkins IBM Champion posted Fri August 01, 2025 12:50 PM

Hi all,

I suspect this is slightly obscure but...

I'm looking to use activity trace to track messages and I would like to add the DLQ into this tracking - rather than set up a separate mechanism just for the DLQ.

However, I have setup a scenario whereby a a remote QM sends a message to QM2 where there is no such queue i.e. it's the channel that's putting the message to the DLQ on QM2. And... I don't see the message in the activity trace. I had hoped to see, perhaps, the channel process in the activity trace. Am I missing it (there's a lot of trace!) or is there any magic incantation to get DLQ/mq processes activity traced.

As I say, I have fallback positions to monitor the DLQ - but I'd rather just use the activity trace if I can. So, any help would be appreciated.

many thanks - as always !

John.

Morag Hughson's profile image
Morag Hughson IBM Champion

I can confirm that the channel does show up in Activity Trace, here' some I generated just now:
- 09:05:22  34092(  2) [ 2832us] :\mqm9400\bin64\amqrmppa.exe MQCONNX 
- 09:05:22  34092(  2) [   62us] :\mqm9400\bin64\amqrmppa.exe MQOPEN NOT.EXIST RC(2085) Unknown object name
- 09:05:22  34092(  2) [   66us] :\mqm9400\bin64\amqrmppa.exe MQOPEN 
+ 09:05:22  34092(  2) [   17us] :\mqm9400\bin64\amqrmppa.exe MQINQ 
  Sat Aug  2 09:05:22 2025
  Context
  MQINQ(
     Connection Id:414D51434D5147322020202020202020FA2B8D6800280040
     Hobj         :4 QMGR(MQG2)
     Sel. Count   :1    
     Selectors    :[0]  2006 MQCA_DEAD_LETTER_Q_NAME
     CompCode     :0    
     Reason       :0     OK
    )
- 09:05:22  34092(  2) [   12us] :\mqm9400\bin64\amqrmppa.exe MQCLOSE 
- 09:05:22  34092(  2) [  504us] :\mqm9400\bin64\amqrmppa.exe MQOPEN SYSTEM.DEAD.LETTER.QUEUE 
- 09:05:22  34092(  2) [   73us] :\mqm9400\bin64\amqrmppa.exe MQPUT SYSTEM.DEAD.LETTER.QUEUE 
- 09:05:22  34092(  2) [  286us] :\mqm9400\bin64\amqrmppa.exe MQCMIT 

Perhaps you did not turn on activity trace until after the channel had started? I turned it on and then recycled my queue manager just to be certain. Long running applications, which I guess amqrmppa would count as, may not pick up the instruction to use activity trace if they are already running.

Cheers,
Morag

John Hawkins's profile image
John Hawkins IBM Champion

Thanks Morag !
You're right and thanks for confirming.

 I was already recycling the QM but there's something about the remote sender channel not stopping or something that means the first message seems to be lost.

QM1 -> QM2 connected and happy. Able to send message and appears on QM2 DLQ.

stop QM2

QM1 sender still says it's running. but no amqrmppa processes around.

restart QM2 receiver channel restarts.

first message from QM1 to QM2 gets lost.

second message appears on the QM2 DLQ. 

Error message in QM1 saying channel ended abnormally (I get that). Maybe it just threw the message away - it was non-persistent.

However, I think I'll note that as "for another day" issue. I have my trace OK in "normal operation" by the looks of it.

thanks for confirming Morag ! (and that's an hour gone on a sunday afternoon ;-)

bruce2359's profile image
bruce2359

Display your Sender channel definition, post it here.

I’ll guess that it’s a FAST channel - the initial value in the SYSTEM.DEFAULT sender channel.  FAST channels deliver non-persistent messages outside a UofW.  Since the destination queue does not exist, amqrmppa will attempt to put the the NP message to the DLQ.  If the message length is greater than the max msg length of the DLQ, amqrmppa will DISCARD the message - no error will be written to system log - working as documented.  

The creating app could request MQRO for exceptions/failures to deliver. 

om prakash's profile image
om prakash IBM Champion

@John Hawkins As you explore activity trace; what about OTEL which helps in tracing message. Is OTEL an option application activity?

bruce2359's profile image
bruce2359

I'd suggest using IBM supplied utility dspmqrte to determine where/how the "missing" NP message traversed from QM1 to QM2.  Please post the results here.

What was the message length of the NP message?  What was the maxmsglength of the DLQ at QM2?

What VRMF on both qmgrs?

John Hawkins's profile image
John Hawkins

Hey Bruce, despite saying I was going to kick this into the long-grass my interest was piqued when you suggested trace route.

Not good news though...

scenario:

1) MQIN.T sending to OpsisDemo - works fine. msg appears on OpsisDemo#DLQ.

2) stop OpsisDemo - sender remains running (expected given that it uses standard default heartbeats and disconnects)

3) restart OpsisDemo - receiver channel does not start (as expected)

4) restart sender channel

5) bye bye message.

This happened even when the sender went into retry mode of its own accord having recognised the receiver was down. That time the message re-appeared on the transmit q, got taken off and then disappeared !!

Here's the trace..

C:\Windows\System32>dspmqrte -q To.OpsisDemo -m MQIN.T -d yes -t high -v all -l yes -rq REPLY.Q
AMQ8653I: DSPMQRTE command started with options '-q To.OpsisDemo -m MQIN.T -d yes -t high -v all -l yes -rq REPLY.Q'.
AMQ8659I: DSPMQRTE command successfully put a message on queue 'OpsisDemo', queue manager 'MQIN.T'.
AMQ8674I: DSPMQRTE command is now waiting for information to display.

--------------------------------------------------------------------------------

Activity:
 ApplName: 'es\IBM\MQ\bin64\dspmqrte.exe'
 ApplType: WindowsNT
 ActivityDesc: 'IBM MQ Display Route Application                                '

 Operation:
  OperationType: Put
  OperationDate: '20250805'
  OperationTime: '11092442'

  Message:
   MsgLength: 180

   MQMD:
    StrucId: 'MD  '
    Version: 1
    Report: DiscardMsg |
            PassDiscardAndExpiry |
            Activity
    MsgType: Datagram
    Expiry: 600
    Feedback: None
    Encoding: X'222'
    CodedCharSetId: Embedded
    Format: 'MQADMIN '
    Priority: AsDef
    Persistence: Persistent
    MsgId: X'414D51204D51494E2E5420202020202049728F6801430240'
    CorrelId: X'414D51204D51494E2E5420202020202049728F6802430240'
    BackoutCount: 0
    ReplyToQ: 'REPLY.Q                                         '
    ReplyToQMgr: '                                                '
    UserIdentifier: 'Admin       '
    AccountingToken: X'16010515000000F5ACDF2FB7EF41B5A9E157FAE903000000000000000000000B'
    ApplIdentityData: '                                '
    ApplType: WindowsNT
    ApplName: 'es\IBM\MQ\bin64\dspmqrte.exe'
    PutDate: '20250805'
    PutTime: '11092442'
    ApplOriginData: '    '
  QMgrName: 'MQIN.T                                          '
  QName: 'To.OpsisDemo                                    '
  ResolvedQName: 'OpsisDemo                                       '
  RemoteQName: 'anything                                        '
  RemoteQMgrName: 'OpsisDemo                                       '

 TraceRoute:
  RouteDetail: High
  RecordedActivities: 1
  UnrecordedActivities: 0
  DiscontinuityCount: 0
  MaxActivities: Unlimited
  RouteAccumulation: None
  RouteForwarding: IfSupported
  RouteDelivery: Yes

--------------------------------------------------------------------------------

Activity:
 ApplName: 'es\IBM\MQ\bin64\runmqchl.exe'
 ApplType: WindowsNT
 ActivityDesc: 'Sending Message Channel Agent                                   '

 Operation:
  OperationType: Get
  OperationDate: '20250805'
  OperationTime: '11102291'

  Message:
   MsgLength: 608

   MQMD:
    StrucId: 'MD  '
    Version: 1
    Report: DiscardMsg |
            Activity
    MsgType: Datagram
    Expiry: 16
    Feedback: None
    Encoding: X'222'
    CodedCharSetId: 437
    Format: 'MQXMIT  '
    Priority: 0
    Persistence: Persistent
    MsgId: X'414D51204D51494E2E5420202020202049728F6804430240'
    CorrelId: X'414D51204D51494E2E5420202020202049728F6801430240'
    BackoutCount: 1
    ReplyToQ: 'REPLY.Q                                         '
    ReplyToQMgr: 'MQIN.T                                          '
    UserIdentifier: 'Admin       '
    AccountingToken: X'16010515000000F5ACDF2FB7EF41B5A9E157FAE903000000000000000000000B'
    ApplIdentityData: '                                '
    ApplType: QMgr
    ApplName: 'MQIN.T                      '
    PutDate: '20250805'
    PutTime: '11092442'
    ApplOriginData: '    '

   EmbeddedMQMD:
    StrucId: 'MD  '
    Version: 1
    Report: DiscardMsg |
            PassDiscardAndExpiry |
            Activity
    MsgType: Datagram
    Expiry: 16
    Feedback: None
    Encoding: X'222'
    CodedCharSetId: Embedded
    Format: 'MQADMIN '
    Priority: 0
    Persistence: Persistent
    MsgId: X'414D51204D51494E2E5420202020202049728F6801430240'
    CorrelId: X'414D51204D51494E2E5420202020202049728F6802430240'
    BackoutCount: 0
    ReplyToQ: 'REPLY.Q                                         '
    ReplyToQMgr: 'MQIN.T                                          '
    UserIdentifier: 'Admin       '
    AccountingToken: X'16010515000000F5ACDF2FB7EF41B5A9E157FAE903000000000000000000000B'
    ApplIdentityData: '                                '
    ApplType: WindowsNT
    ApplName: 'es\IBM\MQ\bin64\dspmqrte.exe'
    PutDate: '20250805'
    PutTime: '11092442'
    ApplOriginData: '    '
  QMgrName: 'MQIN.T                                          '
  QName: 'OpsisDemo                                       '
  ResolvedQName: 'OpsisDemo                                       '

 Operation:
  OperationType: Send
  OperationDate: '20250805'
  OperationTime: '11102291'

  Message:
   MsgLength: 180

   MQMD:
    StrucId: 'MD  '
    Version: 1
    Report: DiscardMsg |
            PassDiscardAndExpiry |
            Activity
    MsgType: Datagram
    Expiry: 16
    Feedback: None
    Encoding: X'222'
    CodedCharSetId: Embedded
    Format: 'MQADMIN '
    Priority: 0
    Persistence: Persistent
    MsgId: X'414D51204D51494E2E5420202020202049728F6801430240'
    CorrelId: X'414D51204D51494E2E5420202020202049728F6802430240'
    BackoutCount: 0
    ReplyToQ: 'REPLY.Q                                         '
    ReplyToQMgr: 'MQIN.T                                          '
    UserIdentifier: 'Admin       '
    AccountingToken: X'16010515000000F5ACDF2FB7EF41B5A9E157FAE903000000000000000000000B'
    ApplIdentityData: '                                '
    ApplType: WindowsNT
    ApplName: 'es\IBM\MQ\bin64\dspmqrte.exe'
    PutDate: '20250805'
    PutTime: '11092442'
    ApplOriginData: '    '
  QMgrName: 'MQIN.T                                          '
  RemoteQMgrName: 'OpsisDemo                                       '
  ChannelName: 'MQIN1.OpsisDemo     '
  ChannelType: Sender
  XmitQName: 'OpsisDemo                                       '

 TraceRoute:
  RouteDetail: High
  RecordedActivities: 2
  UnrecordedActivities: 0
  DiscontinuityCount: 0
  MaxActivities: Unlimited
  RouteAccumulation: None
  RouteForwarding: IfSupported
  RouteDelivery: Yes

--------------------------------------------------------------------------------
AMQ8652I: DSPMQRTE command has finished.

bruce2359's profile image
bruce2359

You've described in narrative form your Sender-Reciever channel failing.  In order to determine why the channel is failing, you need to examine the amqerr01.log file in the errors folder on BOTH qmgrs.  Post the related channel errors logged here.

I asked you to post the Sender channel definition.  Please do so now.  

Francois Brandelik's profile image
Francois Brandelik IBM Champion

Hi John,

Why are you surprised? Did you not see the part for the options of trace messages?:

 Report: DiscardMsg |
            PassDiscardAndExpiry |
            Activity

So what you described would be expected behavior...

As Bruce said, you need to collect the error logs from both ends of the channel to find out what is going on.