I'm facing some performance issues with MQ XMS.NET library.
My design is as below:

1. GET (Session1, Connection1, IN-QUEUE)

 Consumes a message (QUEUEBROWSER) from Inbound and processes it in some other system
2. PUT (Session1, Connection1, OOT-QUEUE)
Put the response from other system into the Outbound
3. DELETE (session 1, Connection 1, IN-QUEUE)
Delete the message from the Inbound
 
The Put and Delete are too slow.. taking 150 ms per message.

Please suggest what could be the reason.

My connection details for Put:

xmsFactoryOut = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
                // Create WMQ Connection Factory.
                connectionFactoryOut = xmsFactoryOut.CreateConnectionFactory();
                // Set the properties

                if (MQMode == (int)Location.Local)
                {
                    LogMessage($"CreateConnectionOut: MQ Location=Local for Adapter [{Name}]", Tracing.LogCategory.Info, Tracing.LogLevel.NormalLow, null);
                }
                else if (MQMode == (int)Location.Remote)
                {
                    connectionFactoryOut.SetStringProperty(XMSC.WMQ_HOST_NAME, MQHostName);
                    connectionFactoryOut.SetIntProperty(XMSC.WMQ_PORT, MQPort);
                    connectionFactoryOut.SetStringProperty(XMSC.WMQ_CHANNEL, MQChannel);
                    LogMessage($"CreateConnectionOut: MQ Location=Remote. MQHostName = {MQHostName} MQPort = {MQPort} MQChannel = {MQChannel}", Tracing.LogCategory.Info, Tracing.LogLevel.NormalLow, null);
                }

                connectionFactoryOut.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
                connectionFactoryOut.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, MQManagerName);
                connectionFactoryOut.SetStringProperty(XMSC.WMQ_QUEUE_NAME, MQQueueNameOut);
                // In case of network issues - reconnect to same queue manager
                connectionFactoryOut.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_OPTIONS, XMSC.WMQ_CLIENT_RECONNECT_Q_MGR);
                connectionFactoryOut.SetStringProperty(XMSC.WMQ_CONNECTION_NAME_LIST, String.Format("{0}({1})", MQHostName, MQPort));
                connectionFactoryOut.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_TIMEOUT, WakeInterval);
                if (!string.IsNullOrEmpty(UserName) && !string.IsNullOrEmpty(Password))
                {
                    connectionFactoryOut.SetStringProperty(XMSC.USERID, UserName);
                    connectionFactoryOut.SetStringProperty(XMSC.PASSWORD, Password);
                }
                //base.LogHandler.Invoke($"Properties set just before the connection {MQHostName}", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugMedium, null);
               
                // Create connection.
                connectionOut = connectionFactoryOut.CreateConnection();
                LogMessage($"CreateConnectionOut: Connection created - {Name}", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
                connectionOut.ExceptionListener = new ExceptionListener(OnExceptionOut);
                // Create session
                sessionOut = connectionOut.CreateSession(false, AcknowledgeMode.AutoAcknowledge);
                LogMessage($"CreateConnectionOut: Session created - {Name}", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);

                destinationOut = sessionOut.CreateQueue(MQQueueNameOut);
                destinationOut.SetBooleanProperty(XMSC.WMQ_MQMD_WRITE_ENABLED, true);
               
                LogMessage($"CreateConnectionOut: Destination created - {Name}", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
                // Create producer
                messagePoducerOut = sessionOut.CreateProducer(destinationOut);
                LogMessage($"CreateConnectionOut: Producer created - {Name}", Tracing.LogCategory.Debug, Tracing.LogLevel.DebugLow, null);
                connectionOut.Start();                

Thanks,Balaji

------------------------------
Balaji Patil
------------------------------

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com
------------------------------

Here is the code for Put and logs:
Logs:

02-15-2023 02:58:10:864
SendOutboundMessage: Start sending Outbound message
-------------------------------

02-15-2023 02:58:10:865
SendOutboundMessage: Before calling send message with 
 MessageID=ID:414d5120524d4956414c2020202020206e6b766365772221 
 CorrelationID= 
-------------------------------

02-15-2023 02:58:10:917
SendOutboundMessage: Outbound message sent with 
 MessageID=ID:414d5120524d4956414c2020202020206e6b766365772221 
 CorrelationID= for 
-------------------------------

For delete:
Logs:

02-15-2023 02:58:10:918
DeleteInboundMessage: Start deleting Inbound message
-------------------------------

02-15-2023 02:58:11:030
DeleteInboundMessage: Inbound message deleted

This is the minimum time.. its takes 150 ms sometimes.

------------------------------
Balaji Patil
------------------------------

Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com
------------------------------

To clarify what Andrew said:

1. You should be using a transacted session for inbound messages
2. You should be keeping the session / connection open and not defining them each time in the loop
3. You should be keeping the outgoing session open
4. If you have different replyto destinations, keep an anonymous producer open and pass it the destination on calling send
5. commit  the inbound session after each successful message.

Example

Open inbound connection and session

Open outbound session

Create consumer and producer

while inbound messages

   send reply

commit inbound session (assuming outbound session is not transacted)

end while

close sessions

close connections

Hope it helps

------------------------------
Francois Brandelik
------------------------------

Original Message:
Sent: Wed February 15, 2023 04:09 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

Here is the code for Put and logs:
Logs:

02-15-2023 02:58:10:864
SendOutboundMessage: Start sending Outbound message
-------------------------------

02-15-2023 02:58:10:865
SendOutboundMessage: Before calling send message with 
 MessageID=ID:414d5120524d4956414c2020202020206e6b766365772221 
 CorrelationID= 
-------------------------------

02-15-2023 02:58:10:917
SendOutboundMessage: Outbound message sent with 
 MessageID=ID:414d5120524d4956414c2020202020206e6b766365772221 
 CorrelationID= for 
-------------------------------

For delete:
Logs:

02-15-2023 02:58:10:918
DeleteInboundMessage: Start deleting Inbound message
-------------------------------

02-15-2023 02:58:11:030
DeleteInboundMessage: Inbound message deleted

This is the minimum time.. its takes 150 ms sometimes.

------------------------------
Balaji Patil
------------------------------


Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com

Hi Francois:

We are using a framework where we have two methods running on different thread:
1. StartProcessing
2. ResponseProcessing

In the StarProcesssing method, I'm reading the message using QueueBrower, and sending to other application.
Other application processes it and sends the response... once the response is back, it calls the method ResponseProcessing.
In the ResponseProcessing method, I read the response and send to outbound and then delete the message from Inbound.

Here I can't use the transacted session as both the above methods are running on different threads.
I should delete the message from Inbound only after receiving the response.
So, the delete must happen in the ResponseProcessing message.

This is the current design.

Please suggest if there is any alternative way.

------------------------------
Balaji Patil
------------------------------

Original Message:
Sent: Wed February 15, 2023 08:27 AM
From: Francois Brandelik
Subject: XMS.NET - Performance issue

To clarify what Andrew said:

1. You should be using a transacted session for inbound messages
2. You should be keeping the session / connection open and not defining them each time in the loop
3. You should be keeping the outgoing session open
4. If you have different replyto destinations, keep an anonymous producer open and pass it the destination on calling send
5. commit  the inbound session after each successful message.

Example

Open inbound connection and session

Open outbound session

Create consumer and producer

while inbound messages

   send reply

commit inbound session (assuming outbound session is not transacted)

end while

close sessions

close connections

Hope it helps

------------------------------
Francois Brandelik
------------------------------


Original Message:
Sent: Wed February 15, 2023 04:09 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

Here is the code for Put and logs:
Logs:

02-15-2023 02:58:10:864
SendOutboundMessage: Start sending Outbound message
-------------------------------

02-15-2023 02:58:10:865
SendOutboundMessage: Before calling send message with 
 MessageID=ID:414d5120524d4956414c2020202020206e6b766365772221 
 CorrelationID= 
-------------------------------

02-15-2023 02:58:10:917
SendOutboundMessage: Outbound message sent with 
 MessageID=ID:414d5120524d4956414c2020202020206e6b766365772221 
 CorrelationID= for 
-------------------------------

For delete:
Logs:

02-15-2023 02:58:10:918
DeleteInboundMessage: Start deleting Inbound message
-------------------------------

02-15-2023 02:58:11:030
DeleteInboundMessage: Inbound message deleted

This is the minimum time.. its takes 150 ms sometimes.

------------------------------
Balaji Patil


Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com

@Andrew Hickson : Its non-persistent.

------------------------------
Balaji Patil
------------------------------

Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com
------------------------------

If it's non-persistent then it's almost certainly not an IO issue as there is no forced IO when handling non-persistent messages.

It is then a surprise that only the two 'update' operations would be slow.

Is this a test or a production system? 75ms is such a big delay that any server side issue would be likely to be visible in an MQ trace on the server hosting the QMgr. 

XMS is implemented as a layer over the MQI, the "GET" above is therefore a browse, which could be streamed (no TCP line turnarounds), while the MQPUT and the "DELETE" (actually a destructive GET of a specific message) would typically each involve a line turnaround. Have you measured the ping time between the client and the server ?

------------------------------
Andrew Hickson
------------------------------

Original Message:
Sent: Wed February 15, 2023 04:12 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

@Andrew Hickson : Its non-persistent.

------------------------------
Balaji Patil
------------------------------


Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com

Hi Andrew:

Below is the Ping result: 

------------------------------
Balaji Patil
------------------------------

Original Message:
Sent: Thu February 16, 2023 04:12 AM
From: Andrew Hickson
Subject: XMS.NET - Performance issue

If it's non-persistent then it's almost certainly not an IO issue as there is no forced IO when handling non-persistent messages.

It is then a surprise that only the two 'update' operations would be slow.

Is this a test or a production system? 75ms is such a big delay that any server side issue would be likely to be visible in an MQ trace on the server hosting the QMgr. 

XMS is implemented as a layer over the MQI, the "GET" above is therefore a browse, which could be streamed (no TCP line turnarounds), while the MQPUT and the "DELETE" (actually a destructive GET of a specific message) would typically each involve a line turnaround. Have you measured the ping time between the client and the server ?

------------------------------
Andrew Hickson
------------------------------


Original Message:
Sent: Wed February 15, 2023 04:12 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

@Andrew Hickson : Its non-persistent.

------------------------------
Balaji Patil


Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com

Unless the client and server are thousands of miles apart then that sounds awfully high.

Your logs showed 52ms for the PUT and if the ping is representative it might nearly all be in the network.

------------------------------
Andrew Hickson
------------------------------

Original Message:
Sent: Thu February 16, 2023 07:01 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

Hi Andrew:

Below is the Ping result: 

------------------------------
Balaji Patil
------------------------------


Original Message:
Sent: Thu February 16, 2023 04:12 AM
From: Andrew Hickson
Subject: XMS.NET - Performance issue

If it's non-persistent then it's almost certainly not an IO issue as there is no forced IO when handling non-persistent messages.

It is then a surprise that only the two 'update' operations would be slow.

Is this a test or a production system? 75ms is such a big delay that any server side issue would be likely to be visible in an MQ trace on the server hosting the QMgr. 

XMS is implemented as a layer over the MQI, the "GET" above is therefore a browse, which could be streamed (no TCP line turnarounds), while the MQPUT and the "DELETE" (actually a destructive GET of a specific message) would typically each involve a line turnaround. Have you measured the ping time between the client and the server ?

------------------------------
Andrew Hickson


Original Message:
Sent: Wed February 15, 2023 04:12 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

@Andrew Hickson : Its non-persistent.

------------------------------
Balaji Patil


Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com

You might try the IP command  traceroute .... 

This gives you the times to each hop on the path to the destination.

------------------------------
Colin Paice
------------------------------

Original Message:
Sent: Thu February 16, 2023 07:01 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

Hi Andrew:

Below is the Ping result: 

------------------------------
Balaji Patil
------------------------------


Original Message:
Sent: Thu February 16, 2023 04:12 AM
From: Andrew Hickson
Subject: XMS.NET - Performance issue

If it's non-persistent then it's almost certainly not an IO issue as there is no forced IO when handling non-persistent messages.

It is then a surprise that only the two 'update' operations would be slow.

Is this a test or a production system? 75ms is such a big delay that any server side issue would be likely to be visible in an MQ trace on the server hosting the QMgr. 

XMS is implemented as a layer over the MQI, the "GET" above is therefore a browse, which could be streamed (no TCP line turnarounds), while the MQPUT and the "DELETE" (actually a destructive GET of a specific message) would typically each involve a line turnaround. Have you measured the ping time between the client and the server ?

------------------------------
Andrew Hickson


Original Message:
Sent: Wed February 15, 2023 04:12 AM
From: Balaji Patil
Subject: XMS.NET - Performance issue

@Andrew Hickson : Its non-persistent.

------------------------------
Balaji Patil


Original Message:
Sent: Tue February 14, 2023 12:48 PM
From: Morag Hughson
Subject: XMS.NET - Performance issue

I see you have some logging in your program. Can you show us some of that output which is presumably where you see that the PUT and the DELETE are taking 150ms. Hopefully this will shown whether or not you are doing something expensive every time which could be done just once at the start.

Cheers,
Morag

------------------------------
Morag Hughson
MQ Technical Education Specialist
MQGem Software Limited
Website: https://www.mqgem.com

Are the messages persistent or non-persistent ?

The two 'update' type calls are reported as slow, which might suggest you are using persistent messages outside of syncpoint.

This might suggest an IO latency issue and you can look at MQ's perception of IO latency using the amqsruaa sample to look at the LOG topics.

In isolation each undate to a persistent object would typically take two IO latencies (reduces to one under concurrent loads) so an IO latency in the region of 75 ms would exlain this time. That would be a horrific IO latency in this day and age, where a few microseconds might be more appropriate.

It would typically be an error to be processing persistent messages outside syncpoint.

On older MQ versions there are further performance issues associated with concurrent processing of large numbers of persistent messages outside of syncpoint.

------------------------------
Andrew Hickson
------------------------------