Tracking Down MQ Clients Poirot Style

For those who like a good Hercule Poirot or Sherlock Holmes detective story, this blog post will look at leveraging some Linux tools and cryptic MQ commands to track a channel instance back to the MQ client application process that is running the channel.  In this example we will assume our MQ queue manager and MQ client application is running on Linux servers.  We will start the investigation from a running SVRCONN channel instance on a queue manager.

Starting With the SVRCONN Channel Instance

We start with running a “DIS CHSTATUS(CLIENT.TO.SERVER) JOBNAME” command on our MQ queue manager, which produces results like the following:

dis chstatus(CLIENT.TO.SERVER) jobname

     2 : dis chstatus(CLIENT.TO.SERVER) jobname

AMQ8417I: Display Channel Status details.

   CHANNEL(CLIENT.TO.SERVER)               CHLTYPE(SVRCONN)

   CONNAME(10.100.10.224)                   CURRENT

   JOBNAME(00007574000000C2)               STATUS(RUNNING)

   SUBSTATE(RECEIVE)                   

The key piece of data we want to work with is the JOBNAME(00007574000000C2).  This is giving us the pid and the tid in hex of the amqrmppa (channel pooling process in our case) program that is running our specific CLIENT.TO.SERVER channel instance.  We decipher the pid and tid of the amprmppa as follows:

0x00007574 = 30068

0x000000C2 = 194

So for our specific CLIENT.TO.SERVER channel instance, the pid is 30068 and the tid is 194.

Using amqrdbgm to Find the Socket File Descriptor

You are probably not familiar with amqrdbgm as it is not a documented command, but it is a tool that comes with distributed IBM MQ.  amqrdbgm is used for looking at the channel cache.  We will use it to find the specific socket file descriptor for our pid 30068 and tid 194.  Here is an example of how to do this from a Linux shell script on our queue manager server.

> amqrdbgm

5724-H72 (C) Copyright IBM Corp. 1994, 2020.

Selected PID(-1) TID(-1) Options()

pid

Enter PID

30068

Selected PID(30068) TID(-1) Options()

tid

Enter TID

194

Selected PID(30068) TID(194) Options()

c

30068.194   CLIENT.TO.SERVER     10.99.16.224

  TCP/IP Socket(16) Blocking(no) RcvTimeout(no)

No matching work structures found

Selected PID(30068) TID(194) Options()

Q

From the above command, we have found that the our CLIENT.TO.SERVER channel instance is using the socket file descriptor 16 for its TCP network connection.

Finding the Remote Port of a Socket With lsof

Now we will use the lsof command to find the remote port for this file descriptor 16 socket for the 30068 pid.  We run the command “lsof -p 30068” on our queue manager server (under the mqm id) and we get the following output.  I have only included some of the data as we are only concerned about the file descriptor 16 of this 30068 pid.

> lsof -p 30068

COMMAND    PID USER   FD   TYPE             DEVICE  SIZE/OFF      NODE NAME

amqrmppa 30068  mqm  cwd    DIR              253,7      4096    262151 /var/mqm

amqrmppa 30068  mqm  rtd    DIR              253,0      4096         2 /

amqrmppa 30068  mqm  txt    REG              253,9      9032    143189 /opt/mqm/bin/amqrmppa

.

.

.

amqrmppa 30068  mqm   14u  IPv4          262444646       0t0       TCP server1:1414->server2:45670 (ESTABLISHED)

amqrmppa 30068  mqm   15u  IPv4          262444652       0t0       TCP server1:1414->server2:41245 (ESTABLISHED)

amqrmppa 30068  mqm   16u  IPv4          265488542       0t0       TCP server1:1414->server2:44414 (ESTABLISHED)

We can see from the above output that the remote port for the TCP connection on socket file descriptor 16 is 44414 and it is located on server2.

Finding the Process that Uses Remote Port 44414 on Server2

For identifying the MQ client application using our CLIENT.TO.SERVER channel instance, we will now go to server2 and find which pid is using port 44414.  You will probably need someone with root access to run this command and be able to get back the data.  From my testing, you have to run this either as root or under the id that owns the 44414 socket.  It is easier to just run it under root.

# run under root

> lsof -i TCP:44414

COMMAND    PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

amqsputc 25744  mqm    5u  IPv4 265488541      0t0  TCP server2:44414->server1:1414 (ESTABLISHED)

We are now able to see that MQ client application amqsputc with pid 25744 on server2 was the one connecting to our specific CLIENT.TO.SERVER channel instance from our queue manager. 

This example was also a direct TCP connection from the MQ client Server2 to our MQ queue manager Server1.  If there was a load balancer in between, we would just be able to tie the TCP connection from our MQ queue manager Server1 to the load balancer server.  However, if you get your network team in involved, you will probably be able to tie the load balancer server back to the MQ client server in a similar fashion as we demonstrated here by chasing the appropriate ports.

Conclusion

This blog post gives a you a technical way to track back a running MQ SVRCONN channel instance to a specific MQ client application when Linux is the server platform.  It just involves using some cryptic MQ commands like amqrdbgm (channel cache program), lsof, and “using the little grey cells”.  A similar approach can probably be done on other platforms by substituting the lsof command for the platform specific equivalent.