MQ

MQ

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  Client pod autoscaling in Kubernetes based on MQ queue depth.

    Posted Wed December 14, 2022 04:49 PM
    Does anyone know about, or perhaps has written, a Kubernetes autoscaler that checks queue depth to know when more queue consumer instances are needed?  Note, this would be scaling of the MQ clients and not the queue managers.  I found this package (https://github.com/ibm-messaging/mq-keda), but looks like it hasn't seen activity in about two years.

    ------------------------------
    Jim Creasman
    ------------------------------


  • 2.  RE: Client pod autoscaling in Kubernetes based on MQ queue depth.

    Posted Thu December 15, 2022 04:48 AM
    Hi Jim
    In general, you have to run your own "Metrics Server" which queries the qmgr for queue depths, then tells Kubernetes to increase/reduce pods..
    What is your Kubernetes strategy - each queue its own container?
    I could imagine that a few of the known MQ 3rd party tool vendors already have a plugin in place that one could use.
    However, if a pod is busy consuming messages, its CPU etc is going up, so you could use the default autoscaler based on CPU usage.

    ------------------------------
    Matthias Jungbauer
    ------------------------------

    Original Message


  • 3.  RE: Client pod autoscaling in Kubernetes based on MQ queue depth.

    Posted Thu December 15, 2022 05:24 AM
    Hi Jim,

    Take a look at: https://keda.sh/docs/2.5/scalers/ibm-mq/

    ------------------------------
    Regards,

    Martin Evans
    IBM MQ Technical Product Manager
    ------------------------------

    Original Message


  • 4.  RE: Client pod autoscaling in Kubernetes based on MQ queue depth.

    Posted Thu December 15, 2022 05:39 AM
    Hi Jim,

    There's an overview and demo published on IBM Developer > https://developer.ibm.com/blogs/scaling-mq-applications-using-keda/ < which might be useful.

    Best regards

    Richard

    ------------------------------
    RICHARD COPPEN
    ------------------------------

    Original Message


  • 5.  RE: Client pod autoscaling in Kubernetes based on MQ queue depth.

    Posted Thu December 15, 2022 08:55 AM
    Thanks, everyone!  Really good information in these replies, especially the short video in the link provided by @RICHARD COPPEN.  We're just starting to look at this sort of dynamic autoscaling.  I hadn't thought about letting the autoscaling take the number of consumers all the way down to zero when the queue is empty, but that makes sense. 

    Queue depth is only one aspect to consider in terms of deciding if more consumer pods are needed.  The consumer might be constrained by some other backend resource (e.g., a database).  Simply bringing up more instances of a consumer in this case would only exacerbate the problem.  We've got a lot to think about before tackling this one.  Any solution will need to take a broader look at the ecosystem before taking action.  I'll look into KEDA to understand how it works and what it provides.


    ------------------------------
    Jim Creasman
    ------------------------------

    Original Message