Figure 1. Monitoring properties of a node of a message flow in the IBM Integration Bus Toolkit


From the “Monitoring” page, monitoring events can be added to the selected node or edited:
Figure 3. Infrastructure provided by IBM Integration Bus for implementing monitoring applications

Business Transaction Monitoring

A “business transaction” is described by the IBM Integration Bus Knowledge Center as “A self-contained business function; for example, the booking of an airline ticket. A business transaction might be implemented as multiple user transactions or activities”. A business transaction may correspond to an IBM Integration Bus message flow or it may encompass several IBM Integration Bus message flows participating in the processing of the same data in sequence.

IBM Integration Bus includes a functionality for monitoring business transactions, named “Business Transaction Monitoring”. The global architecture used by this functionality is the following:

Figure 4. Architecture of the Business Transaction Monitoring functionality of IBM Integration Bus v10

The upstream part of this architecture is the mechanism described above for generating monitoring events as MQ messages and addressing them to MQ topics. The downstream part includes:

• An inner component of IBM Integration Bus which subscribes to the topics in order to receive the monitoring events and record them into tables of a relational database.
• A GUI (Web application), included in the Web User Interface of IBM Integration Bus, enabling administrators to:

First, define the settings of the business transactions to be monitored: Figure 5. Defining the settings of a business transaction to be monitored

In the above screen, administrators define which events are used for monitoring a business transaction and how to correlate and use them to gather monitoring data about a transaction instance.

• Then monitor the in-progress or completed instances of the business transactions:
  

Figure 6. Viewing business transactions

IBM Business Monitor

It is also possible to use IBM Business Monitor to monitor message flows or business transactions executed by IBM Integration Bus. IBM Business Monitor is an independent product which must be purchased in addition to IBM Integration Bus.

The architecture of this solution is the following:

Figure 7. Architecture of monitoring IBM Integration Bus message flows or business transactions with IBM Business Monitor

This approach is fully documented in IBM Integration Bus v10 Knowledge Center

Custom monitoring applications

It is also possible to implement custom applications for monitoring the message flows and the business transactions. Such applications must subscribe to the MQ topics used by IBM Integration Bus in order to receive the monitoring events, then extract the information about the flows from the events and store them somewhere, then provide a GUI to enable the administrators to get access to the information about the flows and perform analysis. The Advanced Monitoring Asset for IBM Integration Bus is a custom monitoring application implemented according to these principles.

Advanced Monitoring Asset for IBM Integration Bus

What is named “business flows” in this part of the article is the same as “business transactions” in the preceding part and in the IBM Integration Bus Knowledge Center. To be more explicit, a business flow means a series of actions which belong to the common processing of a set of data in IBM Integration Bus. It may correspond to an IBM Integration Bus message flow or it may encompass several IBM Integration Bus message flows participating in the processing of the same data in sequence. It might even include some actions performed outside IBM Integration Bus, if they also participate in the same processing and if they are able to generate monitoring events as MQ messages.

The Advanced Monitoring Asset for IBM Integration Bus was designed to monitor IBM Integration Bus message flows and to extend the native Business Transaction Monitoring capabilities of IBM Integration Bus.

Especially:

• Enable the monitoring of flows executed on several integration nodes from a single GUI.
• Provide detailed information about the flows as well as sorting and filtering capabilities.
• Provide statistics about the throughputs, the processing times…
• Remove the need of a “monitoring model” (use a implicit one instead), so that new business flows can immediately be monitored as soon as their settings enable them to publish events and they are deployed.

The Advanced Monitoring Asset is an asset owned by the IBM Cloud Services organization which can be purchased by customers eager to get advanced monitoring capabilities for IBM Integration Bus. An evaluation version of the asset is included as side material of this article and can be downloaded freely (it is limited to retrieving and displaying not more than 30 matching instances at a time when querying the business flow instance data from the database). Full details about how to setup and use the asset are not included in this article (for that purpose, please read the technical documentation also included with the evaluation version as side material). The main intention of this article is to provide a high level view of the philosophy and the functionalities offered by the asset.

Main functionalities

The main functionalities provided by the Advanced Monitoring Asset for IBM Integration Bus are:

• Extract data about the business flow instances executed by IBM Integration Bus from the monitoring event messages generated by the message flows and store them into a database.
• Provide a GUI enabling administrators to query, display and analyze these data.

The GUI part of the asset offers the following functionalities:

• Query the business flow instance data from the database with sorting and filtering them using various criteria (period of time, integration node, integration server, type of business flow, business data) and display the data retrieved as a table.
  

Figure 8. Main panel of the Advanced Monitoring Asset with the results of a query

Display statistics as bar charts about the business flow instances retrieved by a query. Various
statistics are offered:

• About the throughputs by periods of time.
• About the throughputs by nodes/servers and by types of business flow.
• About the processing times by periods of time.
  

Figure 9. Statistics displayed by the Advanced Monitoring Asset

The statistics are displayed as bar charts built using HighCharts, a high quality framework for building business charts developed by the HighSoft company .

Architecture

The architecture is designed as another monitoring application which subscribes to the monitoring events emitted by the IBM Integration Bus message flows and consumes them. As such, it shares the same base and some common features with the “Business Transaction Monitoring” functionality of IBM Integration Bus (introduced in the previous part). However, the Advanced Monitoring Asset subscribes to a different (more generic) topic, stores data into a database with a different structure and the GUI is also different.

Figure 10. Architecture of the Advanced Monitoring Asset

Setup

Enabling the message flows to emit monitoring events

Enabling monitoring is done in almost the same way as described in the first part (“Setting message flows to emit monitoring events”). To access these settings, with the IBM Integration Bus Toolkit, open the message flow with the editor, select a node and retrieve the “Monitor” tab in the “Properties” view.

The only specifics of the Advanced Monitor Asset are that:

• For a common flow instance, it is possible to correlate together events coming from several different integration nodes and servers.
• It is possible to have several start events and several end or failed events for a business flow instance.
• The event name is used to hold the business flow type and the status (in addition to the event name).
  

Figure 11. Defining events for the Advanced Integration Asset

More specifically, the value of that field can contain up to 3 parts separated by ‘#’ characters.
When 3 parts are provided:

• The first part is understood by the asset as the business flow type.
• The second part is understood as the status (“S” for “Started”, “E” for “Ended”, “F” for “Failed” and “P” or any other value for “In process”).
• The third part is understood as the event name. It is not needed for all events to include a business flow type and a status:
• When the business flow type is not included, it is obtained from another event participating in the same business flow instance (at least one of the events must provide it). When several events belonging to a common business flow instance provide contradictory information about the business flow type, the value provided by the last start event takes precedence.
• When the status is not included, “In process” is assumed.
• The events are correlated using the “global transaction correlator” setting only (the other correlators are not taken into account).
  

Figure 12. Correlating events for the Advanced Monitoring Asset

Data taken from the fields of the message processed by the message flow and included in the event are supported. When an event includes data, these data are updated in the database for the business flow instance and they replace entirely the preceding set of data (provided by a previous event of the same instance), if any. When an event does not include data, the preceding set of data recorded for the business flow instance, if any, is left unchanged in the database. Figure 13. Including payload data into the events for the Advanced Monitoring Asset

Activating monitoring of the message flows on the integration server

By default, monitoring is not activated for the message flows hosted by an integration server. An activation operation must be executed for that purpose. This can be done either by using the IBM Integration Bus Web admin interface (in the interface, retrieve the integration server, the application or the flow and, with the mouse, right click on it and select “Start flow monitoring”).

Figure 14. Activating flow monitoring from the IBM Integration Bus Web admin interface

This can also be done using a command. For example, to enable monitoring for a single message flow:

mqsichangeflowmonitoring <integration_node> -c active -e <integration_server> -f <message_flow>

Configuring MQ components to gather monitoring events from multiple sources

When several IBM Integration Bus nodes associated to different queue managers are involved in the configuration, a “target” queue manager which will receive the monitoring events from the various “source” queue managers associated to IBM Integration Bus nodes needs to be created. This “target” queue manager will make the events available to the Advanced Monitoring Asset.

In this “target” queue manager, several objects must be created:

• A local “target” queue which will receive and store the monitoring events and make them available to the Advanced Monitoring Asset.
• A receiver channel which will receive the monitoring events sent by the other queue managers (associated to IBM Integration Bus nodes).

On the other “source” queue managers, associated to the integration nodes hosting the message flows to be monitored, the following objects need to be created:

• A local transmission queue which will be used to send monitoring events to the “target” queue manager.
• A sender channel which will also be used to send events to the “target” queue manager.
• A remote queue which will also be used to send events to the “target” queue on the “target” queue manager.
• A subscription with topic string “$SYS/Broker/<node>/Monitoring/+/+” pointing to the preceding remote queue as destination. This subscription will catch the monitoring events matching this topic and they will be forwarded to the “target” queue manager and queue.
  

Figure 15. MQ infrastructure used to route events to the Advanced Monitoring Asset when several nodes must be monitored together

When one single IBM Integration Bus node and one single queue manager is involved in the configuration, it is not necessary to define a target queue manager, nor channels, nor a remote queue. One single queue and a subscription pointing to it are enough to receive monitoring events and make them available to the asset.

Configuring DB2 or Oracle to store monitoring data

A database with two tables must be configured for the Advanced Monitoring Asset:

The first table will be used to store the monitoring event data.

Figure 16. Table storing the monitoring event data (DB2)

The second table will be used to store the business flow instance data Figure 17. Table storing the business flow instance data (DB2)

Using the GUI

The GUI is a web application which can be accessed using a browser (Firefox or Internet Explorer). It first displays the main menu:

Figure 18. Advanced Monitoring Asset main menu

Querying/listing business flow instances

Using the first action of the main menu (“See all business flows”), the business flow instances matching selection criteria (period of time when the business flow instance executed and/or business data) can be retrieved from the database and displayed as a scrollable table.
The business data used for filtering are those defined as the “Event payload” property of the message flow nodes (when defining the monitoring events to be emitted). Up to three data filters can be used.

Both simple and complex data can be used for filtering:

• For simple data, only the name and value must be specified.
• For complex data, the name must include a path to the single data (for example: “Details/Geography/Item/City”).
  

Figure 19. List of business flow instances

In the list of results, completed business flow instances are displayed in green, while failed business flow instances are displayed in red and in-process (or in-doubt) business flow instances in orange. The columns of the list are: the ID of the business flow instance (defined as the “Global transaction correlator” property in the message flow nodes), the current status (either “Started”, “In process”, “Completed” or “Failed”), the time when the business flow was started, the time when the business flow was ended, the business flow type (defined as the “Event name” property of the message flow nodes), the integration nodes and servers which executed the business flow instance (or part of it), the business data (defined as the “Event payload” property of the message flow nodes). Using the second action of the main menu (“See all failed business flows”), you can limit the query to failed business flow instances only, matching selection criteria, and display them as a scrollable table. Figure 20. List of failed business flow instances

Using the third action of the main menu (“Monitor by integration nodes/servers”), you can first select an integration node, then (optionally) an integration server, then (optionally) a business flow type and finally retrieve the business flow instances matching selection criteria and display them as a scrollable table.
Figure 21. List of business flow instances selected by node, then server, then business flow type

Using the fourth action of the main menu (“Monitor by business flow types”), you can first select a business flow type, then (optionally) an integration node, then (optionally) an integration server and finally retrieve the business flow instances matching selection criteria and display them as a scrollable table. Figure 23. Sorting the list of business flow instances

By clicking on the ID (first column) of a business flow instance in the list, it is possible to display all the events which have been received for this instance as another table below. Figure 24. List of events received for a business flow instance

Displaying statistics about the business flow instances

From the list of business flow instances retrieved by a query, it is possible to display several kinds of statistics.

Figure 25. Links to display statistics

The statistics are displayed as bar charts built using HighCharts, a high quality framework for building business charts.

Statistics about throughputs by periods of time

The first kind of statistics displays information about the numbers of business flow instances processed by periods of time: the lines represent different periods of time and the lengths of the bars represent the number of instances for the periods of time.

Figure 26. Statistics about throughputs by periods of time

Several actions above the bar chart enable you to change the view:

• With “More details (time)” and “Less details (time)”, it is possible to change the period of time used for splitting the data (by seconds, minutes, hours, days, weeks, months or years).
  

Figure 27. Throughput statistics displayed using various periods of time

Conclusion

IBM Integration Bus message flows (or groups of message flows processing the same data) can be monitored in several ways. First, IBM Integration Bus provides an infrastructure for generating monitoring events and addressing them to MQ topics. Next, several applications can make use of these events in order to provide administrators with monitoring information. The first one to be considered is the Business Transaction Monitoring functionality of IBM Integration Bus. When the customer’s expectations go beyond the capabilities of this functionality, other options such as the Advanced Monitoring Asset for IBM Integration Bus should be considered.
The Advanced Monitoring Asset for IBM Integration Bus is an asset implemented by the IBM Cloud Services organization which can be purchased from this organization. It is built on top of the monitoring event infrastructure provided by IBM Integration Bus with an architecture similar the Business Transaction Monitoring functionality. It provides IBM Integration Bus administrators with advanced functionalities such as monitoring business flows executed by different integration nodes or different integration servers from a common GUI, detailed information about the flows, powerful sort and filter capabilities and statistics about the throughputs and the processing times displayed as charts. An evaluation version of the asset is available as side material.

Appendix

Evaluation version

An evaluation version of the asset is provided as side material. Its limitation is that not more than 30 lines can be retrieved when querying the business flow instances from the database.
(Side material .zip file attached)

• A technical documentation in PDF format.
• “IIBAdvMonMDBWeb.war” and “IIBAdvMonGUIWeb.war”: The files to be used to install the two parts of the asset (the Message Driven Bean part and the GUI part). For installing the applications, process as described in the chapter “Installing the asset modules on the application server” of the technical documentation.
• “CreateAdvMonDB_DB2.sql”: A sample DB2 script for creating the DB2 tables needed by the asset. Before running it to create the tables, review its contents and customize it for your environment.
• “CreateAdvMonDB_Oracle.sql”: A sample Oracle script for creating the Oracle tables needed by the asset. Before running it to create the tables, review the contents of this file and customize it for your environment.
• Three example flows ready for monitoring with the asset:

• • “BTM_RETAIL_APP.zip” is a example provided by IBM Integration Bus (slightly adapted for monitoring by the asset). It includes a set of flows designed to work together. Instructions for installing and using the example are provided in the IBM Integration Bus Toolkit: open the Tutorials gallery, select the “Business transaction monitoring” entry, then follow the instructions provided by this entry but, at the “Import projects” stage, import the Project Interchange File (“BTM_RETAIL_APP.zip”) provided with the asset rather than the one provided by the Toolkit.Here are the contents of three messages (also provided as XML files) which can be injected into this example:

    <btm_retail><orderId>2</orderId><requestType>orderItem</requestType><customerNumber>5555</customerNumber><customerName>ChocolateFan</customerName><itemNumber>1234-6789-1212</itemNumber><itemName>Chocolate Chip Cookies</itemName><flows></flows><requestTime></requestTime><processTime></processTime><responseTime></responseTime></btm_retail>
    
    <btm_retail><orderId>2</orderId><requestType>orderItem</requestType><customerNumber>5670</customerNumber><customerName>CoffeeFan</customerName><itemNumber>3529-9999-1937</itemNumber><itemName>Columbian Supreme</itemName><flows></flows><requestTime></requestTime><processTime></processTime><responseTime></responseTime></btm_retail>
    
    <btm_retail><orderId>3</orderId><requestType>orderItem</requestType><customerNumber>4785</customerNumber><customerName>TeaFan</customerName><itemNumber>9876-5342-3847</itemNumber><itemName>Ceylon Black Tea</itemName><flows></flows><requestTime></requestTime><processTime></processTime><responseTime></responseTime></btm_retail>

  • “MultiBrokerTransactionApplication.zip” includes four flows (“MultiBrokerTransactionApplication1”, “MultiBrokerTransactionApplication2”, “MultiBrokerTransactionApplication3” and “MultiBrokerTransactionApplication4”) designed for working in sequence on the same message and able to execute on different integration servers, belonging to a common integration node or to different ones.
    When installing this example, you will have to create the input and output queues used by the flows: “MQTQIn”, “MQTQIn2”, “MQTQInter”, “MQTQIn3”, “MQTQOut”:
    • If one flow and the next one are executed on the same node (either on the same server or different servers), the queue used to pass the messages on can be a plain local queue.
    • If one flow and the next are executed on different nodes, the queue used to pass the messages on must be a remote queue.

    If several nodes are used, you will also have to define a pair of channels and a transmission queue so that the two queue managers involved can communicate and exchange messages.

    Here is the contents of a message (also provided as an XML file) which can be injected into this example:

    <Test><Value>1</Value></Test>

  • “Small BusinessApp.zip” includes a single flow which either succeeds (when the contents of the field “Test\Value” of the processed message is neither null no equal to 0) or fails (when the contents of the field “Test\Value” of the processed message is null or equal to 0).When installing this example, you will have to create the input and output queues used by the flow: “SBFInQ”, “SBFOutQ” and “SBFErrQ”.
    Here are the contents of two messages (also provided as JSON files) which can be injected into this example and which completes with success (the first one) or failure (the second one):

    {"Test": {"Id": "12345", "Value": "1", "Details": {"Geography": [{"City": "Paris", "Country": "France"}, {"City": "London", "Country": "UK"}, {"City": "Berlin", "Country": "Germany"}, {"City": "Madrid", "Country": "Spain"}]}}}
    
    {"Test": {"Id": "12345", "Value": "0", "Details": {"Geography": [{"City": "Paris", "Country": "France"}, {"City": "London", "Country": "UK"}, {"City": "Berlin", "Country": "Germany"}, {"City": "Madrid", "Country": "Spain"}]}}}