MQ

MQ

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  dotnet 6 and XMS library performance issues

    Posted Thu October 05, 2023 06:13 AM

    Greetings everyone.

    I am connecting to MQ server version 09020003 running on Linux. My client is running Managed IBMXMSDotnetClient version 9.3.2.1 (because the latest version has a bug in it connecting via TLS see This thread)

    I am reading messages in bulks of 369 messages (at the moment, controlled by numberOfMessages parameter) in a transaction from source Q. I process the message bulk and ones they have been persisted into Oracle I commit the MQ transaction.

    My client is running in a Linux container inside Kubernetes using mcr.microsoft.com/dotnet/aspnet:6.0 base image. When the source Q has accumulated a few thousand messages and I start my client it starts reading away. Each 369 bulk read takes around one second. Is this bad performance expected because I'm reading in a transaction with a single session? This is my method that reads an x amout of messages. The timeout parameter is one second so it either fills up a buffer of 369 messages or hits a one second timeout if there are no more messages, then it returns whatever is in the message buffer. The selector parameter is filtering messages based on custom header metadata value (message source). The transaction parameter is an instance that contains a single IBM.XMS.ISession that is created like so:

    _connectionWMQ.CreateSession(true, AcknowledgeMode.AutoAcknowledge)

    public List<string> GetMessages(ITransaction transaction, string queueName, int numberOfMessages, int timeout = 15000, string selector = null)
{
    using (_log.BeginScope("GetMessages transaction"))
    {
        _log.LogDebug("QueueName: {queueName}, Number of messages: {numberOfMessages}, Timeout: {timeout}", queueName, numberOfMessages, timeout);
        var messageList = new List<string>();
        int messageCounter = 0;

        try
        {
            using var destination = transaction.Session.CreateQueue(queueName);
            using var consumer = transaction.Session.CreateConsumer(destination, selector ?? string.Empty);

            while (messageCounter < numberOfMessages)
            {
                var message = consumer.Receive(timeout);
                if (message is null)
                {
                    return messageList;
                }
                else
                {
                    var txtMessage = (ITextMessage)message;
                    messageList.Add(txtMessage.Text);
                    messageCounter++;
                }
            }

            return messageList;
        }
        catch (Exception x)
        {
            try
            {
                transaction?.Rollback();
                transaction?.Dispose();
            }
            catch
            {
                // Poorly written IBM library can throw exception during dispose ...
            }
        }
    }
}


    What can I do to improve performance? 



    ------------------------------
    Bjarki Björgúlfsson
    ------------------------------


  • 2.  RE: dotnet 6 and XMS library performance issues

    Posted Thu October 05, 2023 06:13 PM

    I suspect that your performance issue may be related to this part of your description:

    "The selector parameter is filtering messages based on custom header metadata value (message source)"

    MQ is not a Database and does not have indexes for arbitrary message property selection. If you have a deep queue and you are selecting messages based on non-indexed content, then you will pay a hit on scanning through the messages looking for the first one that matches your selector.

    You should be able to test this out by running your application without the filter to see the difference in timing.

    Can you tell us a bit more about your selection criteria, your header metadata and why you need to do bulk processing with filtering, i.e. why you don't just want to process all the messages?

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------

    Original Message


  • 3.  RE: dotnet 6 and XMS library performance issues

    Posted Fri October 06, 2023 04:40 AM

    Hello Morag,

    Thank you for the feedback ♥♥. About this question:

    Can you tell us a bit more about your selection criteria, your header metadata and why you need to do bulk processing with filtering, i.e. why you don't just want to process all the messages?

    I can easily change this setup/design, I will remove the selector and compare performance and report my findings here. 



    ------------------------------
    Bjarki Björgúlfsson
    ------------------------------

    Original Message


  • 4.  RE: dotnet 6 and XMS library performance issues

    Posted Thu February 01, 2024 05:31 AM
    Edited by Bjarki Björgúlfsson Thu February 01, 2024 05:34 AM

    I promised that I would report my findings here, better late than never...
    I think the key here was to re-use the session objects. It's expensive to create sessions.

    • Removing the selector did not result in any notable performance gain
    • Best and final solution was to:
      • Stop processing messages in bulk, removing need for transaction
      • Use the asynchronous message listener pattern with onMessage callback in stead of looping and calling consumer.Receive() 
      • Keep the Session, Destination and Consumer objects alive
      • Simplifies client code a lot and performance is good, scale workers instances to minimize latency. 

    Here is the function I created in my own implementation of Consumer, ListenerResources is a simple IDisposable class encapsulating the Session, Destination and Consumer objects:

            public ListenerResources RegisterMessageListener(string queueName, Action<IMessage> onMessage, string selector = null)
        {
            using (_log.BeginScope("RegisterMessageListener"))
            {
                try
                {
                    _log.LogDebug("Registering listener for {Queue} with selector {selector}", queueName, selector);

                    _connectionWMQ.Stop();

                    var session = _connectionWMQ.CreateSession(false, AcknowledgeMode.AutoAcknowledge);

                    // Open the queue to read from
                    var destination = session.CreateQueue(queueName);
                    var consumer = string.IsNullOrEmpty(selector) ? session.CreateConsumer(destination) : session.CreateConsumer(destination, selector);

                    consumer.MessageListener = new MessageListener((IMessage message) =>
                    {
                        _log.LogDebug("Received message '{Id}'.", message.JMSMessageID);
                        onMessage(message);
                    });

                    _connectionWMQ.Start();

                    //We can not dispose of the resources, otherwise we break the message flow. The caller needs to take care of disposal of these resources.
                    return new ListenerResources
                    {
                        Session = session,
                        Destination = destination,
                        Consumer = consumer
                    };
                }
                catch (XMSException ex)
                {
                    HandleXMSException(ex);

                    throw GetGeneralConnectionExecption(ex);
                }
                catch (Exception e)
                {
                    _log.LogError(e, "");
                    throw new Exception("Unexpected error", e);
                }
            }
        }

    This is how I call it in a DataAccess object:

     public ListenerResources RegisterPaymentListener(Action<string, long> callback)
        {
            return Consumer.RegisterMessageListener
                    (
                        Queue,
                        //Queue,
                        (msg) =>
                        {
                            if (msg != null)
                            {
                                var txtMessage = (ITextMessage)msg;

                                callback(txtMessage.Text, msg.JMSTimestamp);
                            }
                        }
                    );
        }

    And finally going further up the callback chain in my background worker service:

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            ListenerResources resources = default;

            try
            {
                while (!stoppingToken.IsCancellationRequested)
                {

                    resources = _mqReadDA.RegisterPaymentListener(_messageHandlingService.HandleMessage);

                    _logger.LogInformation("Listening for messages.");
                    //Wait for ever until stoppingToken kicks in or an error occures.
                    await Task.Delay(Timeout.InfiniteTimeSpan, stoppingToken);
                }
            }
            finally
            {
                resources?.Dispose();
            }
        }

    This results in one MQ connection per worker.

    Are there any plans for Async Await support for the IBM.XMS managed library?



    ------------------------------
    Bjarki Björgúlfsson
    ------------------------------

    Original Message


  • 5.  RE: dotnet 6 and XMS library performance issues

    Posted Fri February 02, 2024 05:28 AM
    Edited by Francois Brandelik Fri February 02, 2024 05:38 AM

    Hello Bjorn,

    Your response doesn't make much sense because you should still be much faster processing in batch.

    Imagine the following set up:

    Define connection factory,

    Define JMSSession / JMSContext

    Define Consumer

    Loop

        msg = consumer.read(timeout)

        if msg == null and list.size = 0  ) check endcondition and exit or continue

        if msg == null and list.size > 0

          {

             process list

             Session/Context.commit() //& handle exceptions with rollback

             continue

             }

         Add message to List

        if (!List full ) {

          continue

        } else {

          Process List

          session/context.commit() // & handle exceptions + rollback

       }

       Check endcondition

    end loop

    close consumer

    close Session/Context

    as for the selector, I would move the message matching to a different queue depending on whether the selector is satisfied or not.

    The bulk process would then not need a selector.

    Remember the most expensive operation is creating the connection and session / JMSContext.

    So creating that in every bulk loop is counter productive and would explain your performance: => you're shooting yourself in the foot...

    So every bulk loop takes 1 second because for every bulk loop you create a new connection / session.



    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message


  • 6.  RE: dotnet 6 and XMS library performance issues

    Posted Fri February 02, 2024 07:17 AM

    Thanks for the feedback Francois

    Yes you are right and as I stated in my reply the creation of sessions is the expensive part. As for the bulk processing in my case single message processing is performing well enough, the whole message process (readQ, readSourceDB, businesslogic, writeTargetDB) takes around 4.5 milliseconds on average measured from the time the message entered the Q until it has been commited to target db.

    In my case it is important that messages are not waiting on the Q for long and are delivered in same sequence, inbound traffic on the queue varies, sometimes there is not much traffic then a single message processing approach is better. Furthermore it simplifies the code.

    So I'm looking at these trade offs:

    Bulk processing
    Pros:

    • Single worker can handle all load with one connection to QManager(messages do not accumulate in the Q, client consumer keeps up with producer)
    • Less network roundtrips to source and target DB and fewer queries/bulk inserts
    • Possibly better resource usage on MQ server?

    Cons:

    • More complex, (optimum bulk size, max wait time when traffic is slow and bulk size not filled quickly etc.)
    • Due to message delivery in sequence requirements, scaling workers/consumers can pose problems (but scaling is probalby not neccecary)
    • Message processing time long when traffic is slow.

    Single message processing
    Pros:

    • Simpler code
    • Scalable
    • Even message processing time
    • Delivery sequence should not pose problems

    Cons:

    • More traffic on source and target db
    • More network roundtrips
    • Need more consumer workers/connections to QManager to keep up with producer. (4 consumer in my case can keep up with producer)

    In my case single message processing is the way to go since network roundtrips do not seem to be such a big deal in terms of latency.

    Regards,
    Bjarki (not Bjorn)



    ------------------------------
    Bjarki Björgúlfsson
    ------------------------------

    Original Message


  • 7.  RE: dotnet 6 and XMS library performance issues

    Posted Mon February 05, 2024 02:29 AM
    Edited by Francois Brandelik Mon February 05, 2024 02:32 AM

    Bjarki,

    Looks like I am not at all getting your following con for batch processing:

    • Message processing time long when traffic is slow.

    The maximum time it would take to process the message is:

    • mq wait time + processing time of the batch. You process the batch whether it is full or not when you receive a null message (Reason Code 2033)
    • Worst case scenario, you are receiving very sparse message and your processing time for all messages in the batch would be:
      • mq wait time * nbr of messages in the batch + processing time of the batch
        In this case I would expect messages to trickle in, just at the edge of the mq wait time, so that you never get a null message... In which case you could limit the time to live of a batch: i.e. if the batch is not full but more than x time has passed since the start of the batch, process the batch even though it it not full...

    If your messages truly arrive in batches that build queue depth, a batch processing architecture is faster than an online one.

    Hope it helps.



    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message