IBM MQ has been reliably and securely transferring data for over 30 years, in that time there have been many features added and integrations with other products created. IBM Watsonx.ai enables users to tune and guide models with their data to meet any need. With IBM MQ being a handler of data and Watsonx.ai being able to process that data there is an obvious opportunity for users of IBM MQ to extract data flowing through IBM MQ to send to IBM Watsonx.ai for further processing. To extract data from IBM MQ without effecting existing message flows we can utilize the IBM MQ Streaming Queues feature.
In this blog post we will explore how combining IBM MQ’s streaming queues with IBM Watson X could enable you to gain insight into the data flowing through your IBM MQ infrastructure, enabling you to make business decisions or create unique experiences for your customers. We will first look at an IBM MQ only option before briefly looking at an approach that combines IBM MQ with another IBM product to perform the same task without the need to produce and maintain your own code.
Scenario
In this scenario I created a simple setup where one bank was sending SWIFT MT103 messages to another via a single queue on an IBM MQ queue manager. To send SWIFT MT103 messages I wrote a demo application, although as I’m no expert on SWIFT messaging I anticipate it is not quite accurate. However, for this scenario It is sufficient. I added a streaming queue to the queue to start siphoning messages to another queue. There I wrote a small application that consumed the messages from the streamed queue, converted them into a prompt that was then sent to and executed by IBM Watsonx.ai. The result to the prompt was then printed out.
The goal was to take those SWIFT MT103 messages and create a simple, human readable summary of the transaction including who sent the money, who received it, how much was sent, what currency, when it was sent and finally the reference of the transaction.
In a real-world situation SWIFT MT103 messages are not sent in plain text, they are signed and encrypted to ensure that private transactions cannot be intercepted or edited. I also focussed on a quite simple version of a MT103 and ignored any process that exists around them. However, for a proof of concept, this scenario creates a starting point and concept that can be built on.
Setting up MQ
For this scenario, the IBM MQ configuration is simple. We need to create the queue for the banks to share and the streaming queue to place the message duplicate. We also need listener, channels, and security for connecting client applications, but this configuration is not covered here.
I ran the following MQSC commands to define the two queues:
DEFINE QLOCAL(‘StreamQ’)
DEFINE QLOCAL(‘LocalQ’) STREAMQ(‘StreamQ’)
Configuring IBM Watsonx
For this scenario I looked at the Watsonx prompt lab to generate a prompt that when passed a MT103 message would generate the desired output. Once logged in and having created my sandbox project I clicked on “Open Prompt Lab”:
In the prompt lab I switched to the “Structured” view, I then spent time providing different Instructions, examples, and inputs until I got the output I wanted. I eventually settled on an instruction of “The following is a SWIFT MT103 message. Summarize it telling me the name of the sending account holder, receiving account holder, money sent, transaction date and currency. Do not provide additional information.“ and three examples. This regularly would give me a summary response of: “ON <date>, <sender name> with account number <Sender account number> sent <amount> <currency> to <receiver name> with account number <receiver account number. The reference was <reference>.”
I found I did not need to change any of the model parameters or the model that was in use from their defaults. Currently the default model is granite-13b-chat-v2.
I then opened the “view code” menu and copied the curl command out for later use:
Before finishing, I created an apikey linked to my account and Watsonx instance so that I could later generate the access token needed to run the curl command. I followed the IBM docs for creating credentials for programmatic access.
Creating the Bridge application
The concept for the bridge application was also simple, I needed an application that would convert an apikey into an access token and then get a message from the streaming queue, wrap it in the prompt I created in the previous section, send it to Watsonx.ai and print out the response. To create the MQ interaction part I utilized the JMS samples provided with IBM MQ. But to create the code that contacted IBM Cloud and IBM Watsonx.ai I used Watson Code Assistant to convert curl commands into java code. I found that WCA enabled me to create the bridge application very quickly as the code that it produced needed minor tweaks to add in the right values. For example, to convert an IBM Cloud apikey into an access token you can use the following curl command:
curl -X POST 'https://iam.cloud.ibm.com/identity/token' -H 'Content-Type: application/x-www-form-urlencoded' -d 'grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=MY_APIKEY'
Using the following prompt in WCA generated the following code that I could adjust and insert into my code. Once I had finished writing the application I had a logic loop that did the following:
- Made a HTTP POST call to IBM Cloud IAM to convert my apikey into an access token.
- Connected to the queue manager and consumed a message from the streaming queue.
- Added the message to the prompt from earlier and sent this as a HTTP POST call to the IBM Watsonx.ai service.
- Printed the response from the HTTP POST call to standard out and then looped to step 2.
You can view the code for the bridge program I created here: https://github.com/parrobe/MQ-watsonx-demo
Trying it out
With IBM MQ configured, the prompt and credentials for Watsonx.ai created and the bridge program written I started the banking applications and bridge program. Immediately I started getting summaries of the MT103 messages that were sent between the two bank applications:
Additional Concepts
In this scenario I focussed entirely on passing messages from IBM MQ into the prompt engine of IBM Watsonx.ai. However, there are other use cases for IBM MQ and IBM Watsonx. For example:
- Classifying message data flowing through IBM MQ to build a picture of what data flows where.
- Analysing messages to spot out of place or potentially fraudulent messages.
- Using IBM Watsonx.data to create AI-powered data insights from the data flowing through IBM MQ.
- Extract IBM MQ Instrumentation events passing them to IBM Watsonx for further analysis and pattern recognition.
If you are wondering whether Watsonx can be combined with the other aspect of messaging, events, then the answer is yes! @Dale Lane published a blog post detailing several examples of how you can combine IBM Event Automation with IBM Watsonx to process event messages. You can read that blog post here.
An alternative to a custom bridge program
IBM Event Automation enables you to connect to Watsonx without the need to write any code. While IBM Event Automation manages events and IBM MQ manages messages it is possible for these two technologies to be connected and exchange data. This opens the possibility to avoid the need for a bridge program extracting messages from IBM MQ to pass onto Watsonx as we can use IBM Event Automation instead.
Combining IBM MQ and IBM Event Automation is possible using the IBM MQ Kafka connectors, @Dale Lane wrote about using streaming queues to create an auditable copy of a message. This could be adapted to feed into IBM Event Automation and then onto Watsonx. This results in a potential scenario that looks like:
Conclusion
In conclusion, thanks to IBM MQ streaming queues functionality and the ability to quick get started with IBM Watsonx.ai by sending requests via HTTP POST calls it is easy to extract data from your IBM MQ network and process it within IBM Watsonx. If writing code to interact with Watsonx is not possible then an alternative is to utilize IBM Event Automation.
Both IBM MQ and IBM Event Automation contain data that is valuable and relevant to you; by utilizing IBM Watsonx to process the data you can gain greater insights that may be missed through traditional processing. This in turn could enable you to provide better levels of service to your customers or find an advantage over your competitors.
References