MQ

 View Only
Expand all | Collapse all

MQ plugin/modules for Ansible

  • 1.  MQ plugin/modules for Ansible

    Posted Tue July 21, 2020 10:38 AM
    Is there any focus on the development of an MQ plugin/modules for Ansible?

    It is known that Ansible is being adopted by larger enterprises to automate maintenances and/or repetitive tasks, and there are a large number of MQ admin tasks that could benefit from that.

    Nowadays, when we want to automate something related to MQ, we depend on shell scripts to call MQ commands (strmqm, endmqm, etc) or run MQSC script thru runmqsc.

    For example, having automation in Ansible, to call a shell script, which will call runmqsc, to run an MQSC command to suspend a QM from a cluster, before stopping the QM does not sound an elegant solution!

    ------------------------------
    Renato Zancoper
    DataPower & MQ Admin
    IBM
    ------------------------------


  • 2.  RE: MQ plugin/modules for Ansible

    Posted Wed July 22, 2020 11:56 PM
    Hello Reanto,
    We are using Ansible to automate MQ tasks across a fleet of AIX, Linux and Windows servers, to create and configure MQ queue managers, MFT agents, MQ clients.
    There are no MQ specific modules that I am aware of.
    The Ansible 'command' and 'win_command' modules allow MQ command lines to be run directly, including switching to the UNIX mqm user if desired. However this is not suitable if any sort of MQ specific decision logic or processing or verification needs to be done on the remote server. A practical solution is for Ansible to upload a script to the remote server, and then run it using the command module, with appropriate arguments. As you know, the output is returned to the Ansible controller in a JSON structure, which can then be saved for later reference.
    HTH,

    ------------------------------
    Glenn Baddeley
    Senior Middleware Software Engineer
    Coles Supermarkets Australia Pty Ltd
    ------------------------------



  • 3.  RE: MQ plugin/modules for Ansible

    Posted Thu July 23, 2020 10:14 AM

    Hi Glenn,

    Yes, I am aware of Ansible 'shell'/'command' modules, but as I said, to take MQ down for simple maintenance, we would have Ansible triggering a shell script, which would trigger 'runmqsc', providing a MQSC script suspend the QM from the cluster, and trigger 'endmqm' ... I already have a ksh script for it !!! ... 

    Looking to the Ansible documentation, I can see modules for a number of different databases (Postgresql seems to be the more complete one), but for Messaging, there is just a limited module for RabbitMQ ...

    Now that IBM and RedHat are a 'couple' would be amazing if the IBM products get integrated with RedHat tools in a more 'native' way!



    ------------------------------
    Renato Zancoper
    DataPower & MQ Admin
    IBM
    ------------------------------



  • 4.  RE: MQ plugin/modules for Ansible

    Posted Wed January 04, 2023 01:22 PM
    We are beginning to scratch this now. 

    Would you be able to share some of what you have developed?

    I'm learning this and would appreciate any help I can get here.

    ------------------------------
    Andrew Robert
    ------------------------------



  • 5.  RE: MQ plugin/modules for Ansible

    Posted Thu January 05, 2023 09:31 AM
    I am using ansible to manage MQ running on AIX, Linux and MQ Appliances for all type of deployments; Standalone, Multi-instance & RDQM deployed on-prem, AWS and Azure.  Even though you can push scripts and run them on remote servers, it's not the "ansible" way.

    I execute each MQ command as its own task in the playbook. This gives me full access to the output produced and any return code generated.  This output can be evaluated, and all decisions can be done in the playbook.    Coding each action in the playbook give you full transparency to what is going on.  This also allows you to build in idempotency where scripts do not or are hard to prove.  

    As for building tasks to do these actions, anyone can develop custom tasks and either keep the private for your own use or build a community module and submit it for public use.  


    ------------------------------
    Rob Simons
    ------------------------------



  • 6.  RE: MQ plugin/modules for Ansible

    IBM Champion
    Posted Thu January 05, 2023 04:58 PM
    With REST available for IBM MQ, you can leverage API call and control most of the MQ Admin activity.

    • Build a Gateway Queue Manager which acts as entry point
      • Enable MQWEB with Auth, etc..
      • Add all the queue managers this endpoint shoudl talk to
    • Now using the REST URL, manage.
    You can do it using PY/Ansible or anu other programming language.
    github actions/gitlabCi etc..all will work.

    For MQ Appliance, use the web URL REST endpoint and simialr manage the appliance. The queue managers can be managed using the above gateway queue manager.

    Parse all output as JSON and take action on failure/error.

    ------------------------------
    om prakash
    ------------------------------



  • 7.  RE: MQ plugin/modules for Ansible

    Posted Fri January 06, 2023 10:02 AM

    These two links seem to suggest IBM is moving in the "right" direction:

    https://doyoumq.com/podcast/introduction-to-mq-ansible/

    https://konnekted.io/@mq_ansible

    But I cannot see any update since a while nor there seems to be a way to provide any suggestion. It seems clear there are multiple people (myself included) who are implementing this their own way. Is the project above still alive and kicking?



    ------------------------------
    Fabio Nuzzi
    Amadeus
    ------------------------------



  • 8.  RE: MQ plugin/modules for Ansible

    Posted Fri January 06, 2023 10:12 AM

    A bit ham handed but if you have the queue managers started via systemd, you can do something like this:

     

    - name: Start MQ services

         hosts: mq_inventory

         remote_user: mqm

         become: true

         become_method: sudo

         gather_facts: true

     

         tasks:

         - name: Start MQ service

           ansible.builtin.systemd:

              name: "{{ item }}"

              state: started

           with_items:

              - 'mq@QMGR1.service'

              - 'mq@QMGR2.service'

     

    The mq_inventory is pulled from your inventory file.

     

    If your queue manager is bound with an ACE broker, you can start both at the same time.

     

    Description: MFS Logo

    Andrew Robert, MQ Architect, Middleware Application Support
    T: 617-954-5882  M: 508-925-0831  E: arobert@mfs.com
    MFS Investment Management
    111 Huntington Avenue, Boston, MA 02119

     

     

     MFS Email system made the following annotation
    ---------------------------------------------------------------------------------------------------------------------------------------
    This email communication and any attachments may contain proprietary, confidential, or privileged information. If you are not the intended recipient, you are hereby notified that you have received this email in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. The sender does not waive confidentiality or any privilege by mistransmission. If you have received this email in error, please notify the sender immediately, delete this email, and destroy all copies and any attachments.





  • 9.  RE: MQ plugin/modules for Ansible

    Posted Mon January 09, 2023 04:58 PM
    There are basic capabilities to interact with IBM MQ queues on z/OS available at:
    z_ansible_collections_samples/zos_subsystems/mq at main · IBM/z_ansible_collections_samples · GitHub

    Here you can use ansible to create queues, alter queues, delete queues and display status.

    Ansible is ideally used for use cases on z/OS that push changes to the environment such as a configuration, but another relevant use case might be using ansible to check the status of queue managers' dead letter queues then sending an alert based on whether or not there is a non-zero queue depth.

    For z/OS-ansible interaction, the z/OS environment must have both ZOAU and Python installed

    ------------------------------
    Dorothy Quincy
    ------------------------------



  • 10.  RE: MQ plugin/modules for Ansible

    Posted Tue January 10, 2023 09:50 AM
    Edited by Portia Melita Fri January 13, 2023 09:33 AM

    Hello Renato,

    Following on from #7 by Fabio Nuzzi​, the MQ-Ansible project is soon to be externalised (NB: the exact timing is undecided at the moment). So far, we have implemented the following, which is tested on the Ubuntu OS:

    • Ansible Roles to fully automate MQ installation process, and Web Console setup.
    • Ansible module 'queue_manager', which allows:
      • Defining and Controlling QMGRs (Create/Start/Stop/Delete)
      • Running MQSC for basic configuration


    (e.g in your case, MQ-Ansible would replace the use of the separate 'shell' and 'command' modules with just the 'queue_manager' module.

    Future features will be added in the coming months.

    I hope this helps!

    Kind Regards,
    James Page

    ------------------------------
    James Page
    ------------------------------



  • 11.  RE: MQ plugin/modules for Ansible

    Posted Thu January 19, 2023 06:44 AM

    Hello Renato,

    Further to this, the MQ-Ansible project is now external on github.com below:

    https://github.com/ibm-messaging/mq-ansible

    Please feel free to get in touch if you have any further questions.

    Kind Regards,

    James Page



    ------------------------------
    James Page
    ------------------------------



  • 12.  RE: MQ plugin/modules for Ansible

    Posted Thu July 27, 2023 03:05 PM

    Hey guys!

    Is there any way to clear a queue that is currently in use with Ansible? In my company, sometimes we need to access the Windows server, run "IBM MQ Explorer", access the Queue Manager, select the queue, run "Clear Messages..." and check "Clear with MQGET API calls". This process is a bit time consuming and I'm trying to do it remotely with Ansible.

    I looked a lot about this and the only way I saw was to create an application to use MQGET API to accomplish this task. I don't have the knowledge in Java application development, C#/C++ and etc to create this, so I thought there might be something ready to do this task in Ansible.



    ------------------------------
    Roger Windberg
    ------------------------------



  • 13.  RE: MQ plugin/modules for Ansible

    Posted Fri July 28, 2023 06:32 AM

    Hi Roger, if you're looking to emulate what MQ Explorer does, you could use the amqsget sample program to destructively get messages from the queue.

    https://www.ibm.com/docs/en/ibm-mq/9.3?topic=programs-running-amqsget-amqsgetc-samples

    If you want to run the sample program on a different host to the queue manager, you'd need to use the amqsgetc client-mode equivalent program, and configure the MQSERVER environment variable or use a CCDT file to provide details of the queue manager's SVRCONN channel to use as described in the link above.

    The content of each message consumed will be output to stdout, so you can redirect to /dev/null if you have no interest in the message data itself (although it may be helpful to keep an amount of audit log of the messages removed, depending on the scenario).

    It's also worth noting that this approach won't work if another application already has the queue open for INPUT_EXCLUSIVE (the same is true of the MQ Explorer approach you've described).



    ------------------------------
    Chris Leonard
    ------------------------------



  • 14.  RE: MQ plugin/modules for Ansible

    Posted Sun July 30, 2023 07:34 PM

    I think the queue is not open in "INPUT_EXCLUSIVE" mode because the process I described in MQ Explorer works fine, it clears the queue. I tried using amqsget but I get an error saying that the queue is already open.

    Searching the internet I saw that I can only clear the queue that is already in use by creating an application that uses MQGET API calls to perform this task. I requested AI support to understand how to do this since I don't have much programming knowledge and the suggestion was to create a Python program with the following code:

    import pymqi
    import CMQC
    
    queue_manager = 'MY_QUEUE_MANAGER'
    channel = 'SYSTEM.DEF.SVRCONN'
    host = 'localhost'
    port = 1414
    queue_name = 'MY_QUEUE_NAME'
    
    qmgr = pymqi.connect(queue_manager, channel, f"{host}({port})")
    
    queue = pymqi.Queue(qmgr, queue_name)
    queue_open_options = CMQC.MQOO_INPUT_AS_Q_DEF | CMQC.MQOO_FAIL_IF_QUIESCING
    
    try:
        while True:
            try:
                message = queue.get()
                print("Message cleared: " + message.decode())
            except pymqi.MQMIError as e:
                if e.comp == CMQC.MQCC_FAILED and e.reason == CMQC.MQRC_NO_MSG_AVAILABLE:
                    print("Queue is empty.")
                    break
                raise e
    finally:
        queue.close()
        qmgr.disconnect()

    My question is: Do you think it might work? Is there even a way to remove messages from the queue with it in use?



    ------------------------------
    Roger Windberg
    ------------------------------



  • 15.  RE: MQ plugin/modules for Ansible

    Posted Mon July 31, 2023 04:00 AM

    I wonder if the queue in question has DEFSOPT(EXCL) in its attributes?

    The amqsput utility uses MQOO_INPUT_AS_Q_DEF in its open options (ie. inherit the value of DEFSOPT), whereas MQ Explorer explicitly sets MQOO_INPUT_SHARED when opening a queue to clear it.

    If the application specifies MQOO_INPUT_AS_Q_DEF and the queue is configured with DEFSOPT(EXCL), the MQOPEN call will fail if any other processes have the queue open for input. The Python sample you provided also uses MQOO_INPUT_AS_Q_DEF, and so would see the same behaviour in this scenario.

    If DEFSOPT(EXCL) is set on your queue, and you're confident that it's not opened exclusively by another process, try using MQOO_INPUT_SHARED by providing explicit open options to amqsget as follows:

    amqsget [queue] [queueManager] 8194

    (8194 is the decimal value of  MQOO_INPUT_SHARED + MQOO_FAIL_IF_QUIESCING)

    If opening the queue with an explicit INPUT_SHARED still fails with object in use, that would suggest the other application does have exclusive access to the queue. In that case there's no way to clear the queue underneath that application, other than disconnecting it from the queue manager.

    As an alternative, you haven't said why or how often you need to clear the queue. If you're clearing it regularly for housekeeping purposes, you could consider using the CAPEXPRY feature to ensure that un-retrieved messages are expired from the queue after a configured interval. See https://www.ibm.com/docs/en/ibm-mq/9.3?topic=mqmd-enforcing-lower-expiration-times



    ------------------------------
    Chris Leonard
    ------------------------------



  • 16.  RE: MQ plugin/modules for Ansible

    IBM Champion
    Posted Tue August 01, 2023 02:44 PM

    Hi Roger,

    I've posted many sample programs on my site at: https://capitalware.com/mq_sample_code.html

    A couple of C programs you might be interested in:

    • ClearQ.c will attempt to clear a queue in 2 different ways: 1st, by issuing the ClearQ PCF command & if that fails, 2nd consume (destructive get) all messages on the queue using a zero-byte message buffer.
    • EmtpyQ.c will consume (destructive get) all messages on the queue using a zero-byte message buffer.

    Java program:

    • EmtpyQ.java will consume (destructive get) all messages on the queue using a zero-byte message buffer.

    Any of these programs will be better than using the MQ sample amqsget.

    later

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London ON Canada
    https://capitalware.com
    ------------------------------



  • 17.  RE: MQ plugin/modules for Ansible

    IBM Champion
    Posted Mon July 31, 2023 06:24 PM

    The mechanism of clearing the queue using repeated MQGET calls is correct for an open queue, but amqsget is not the right application as it has a number of limitations (especially around message size).

    Your (AI) approach may work, but I'm not a python developer so I can't really comment on it.

    However, there is a supplied application which would do the job without you needing to write code.

    dmpmqmsg has a destructive GET option, and if you don't want to save the messages, you can send the output to /dev/null (or equivalent) either with -f or just by leaving -f off completely.

    Something along the lines of

    dmpmqmsg -m QMGR -I EMPTY.THIS.QUEUE -f /dev/null

    or just 

    dmpmqmsg -m QMGR -I EMPTY.THIS.QUEUE

    You could use the -t option to specify how many messages to get in each unit of work (transaction).

    A quick summary of options is available with...

    /opt/mqm/bin/dmpmqmsg -?

    It is documented in the manuals at 

    https://www.ibm.com/docs/en/ibm-mq/9.3?topic=reference-dmpmqmsg-queue-load-unload

    Regards



    ------------------------------
    Neil Casey
    Senior Consultant
    Syntegrity Solutions
    Melbourne, Victoria
    IBM Champion (Cloud) 2019-22
    ------------------------------



  • 18.  RE: MQ plugin/modules for Ansible

    Posted Wed August 02, 2023 06:41 AM

    Hi All, 
    I appreciate you all have this covered.

    FYI, I wanted to add what we state about the CLEAR QLOCAL command in the MQ docs: 

    If an application has this queue open, or has a queue open that eventually resolves to this queue, the command fails. The command also fails if this queue is a transmission queue, and any queue that is, or resolves to, a remote queue that references this transmission queue, is open.

    Regards .. Mayur



    ------------------------------
    Mayur RAJA
    ------------------------------



  • 19.  RE: MQ plugin/modules for Ansible

    Posted Wed August 02, 2023 09:54 PM

    The dmpmqmsg application works fine.

    Thank you!



    ------------------------------
    Roger Windberg
    ------------------------------