Explore the new features in IBM Integration Bus version 10.0.0.6

Integration Bus has been able to invoke external REST APIs for a long time now, but interacting with REST APIs whose interface is well described using a Swagger document (Open API specification) has just got a whole lot easier. The new REST Request node issues a synchronous HTTP request to a REST API and then waits for its response. The node is configured by dragging and dropping an operation on to the message flow canvas, or on to the connection between two existing nodes in a message flow. Swagger documents can originate directly from your file system, from a referenced project, can be retrieved from a URL using HTTP or HTTPS or you can configure the REST Request node manually using a wizard.
The REST Request node has a wide variety of options providing the capabilities you’d expect – for example message parsing of responses from APIs (support for JSON, XMLNSC, BLOB domains etc.), securing its behavior (HTTPS encryption settings, and providing a security identity using mqsisetdbparms) and compression settings. The node also has rich settings for configuring whereabouts in the logical message tree it should expect to receive data and whereabouts it should return data when invoking APIs. This means that it is easy to use multiple REST Request nodes in a message flow to interact with a chain of APIs, without requiring intervening transformation nodes between each invocation. The node also provides a large number of dynamic overrides using the LocalEnvironment tree, so you can easily cope with complex scenarios requiring different behavior from the node on a message by message basis. A new category has also been added to Activity Log for easily viewing the REST Request nodes’ activity.

This pair of new nodes are similar to the REST Request node in that they allow a message flow to issue a REST request to an external REST API, but these nodes behave asynchronously and allow control to be returned to the message flow without waiting for a response from the API.  Ultimately, the response is received by a REST AsyncResponse node, which can be placed in a separate message flow altogether, but it must be deployed to the same integration server.  The nodes are used as a pair and correlate responses with the original request using a unique identifier which is specified on both nodes using the Toolkit. The REST Async nodes use an imported Swagger document which can be in either JSON or YAML format, and which you can import from the Toolkit workspace, the file system or from a URL.

The new LoopBackRequest node allows flow developers to easily create, retrieve, update and delete data records in external systems. This is done using LoopBack connectors which you can either write yourself or download from a large open source catalog. Common connectors that we expect to make popular usecases will be interaction with NoSQL databases such as MongoDB, Cloudant and PostgreSQL.  LoopBack (loopback.io) is an open source Node.js framework which can be used for authoring connectors. IBM Integration Bus has embedded Node.js capabilities in our runtime for a few fix pack releases now, and the new LoopBackRequest node adds to this area of our codebase and also provides the npm tool for the first time.  The npm tool allows users to easily download and install LoopBack connectors which others have already written.  LoopBackRequest nodes are configured to use a specific data source stanza from the LoopBack datasources.json file, and invoke create, retrieve, update or delete operations on a model instance for that data source.  Optionally you can also provide a LoopBack JSON model definition if your connector requires it – in order to provide default values for particular fields for example.  As existing IIB users might expect, the LoopBackRequest node uses security identities in conjunction with identities defined using the mqsisetdbparms command, and also provides dynamic override capabilities using the LocalEnvironment tree.

IIB introduced the concept of a REST API when version 10 became generally available in 2015. REST APIs provide a specialized form of an application which exposes integration logic in message flows so that it can be invoked by HTTP clients who see the universe as a series of resources that can be created, retrieved, updated or deleted.  When first introduced, REST APIs could only be deployed in conjunction with the HTTP listener which is embedded within an IIB Integration Server, but as of version 10.0.0.6 of Integration Bus we are introducing support for REST APIs to also be deployed to the IIB runtime to use the node wide HTTP listener. As part of these changes we have also enabled CORS support in the node wide listener as well.

MQ version 9 is the latest major release of that product which became generally available on 2nd June 2016 (https://www.ibm.com/developerworks/community/blogs/messaging/entry/IBM_MQ_9_0_is_now_available?lang=en), just after IIBv10.0.0.5 was delivered.  So, IBM Integration Bus version 10.0.0.6 becomes the first IIB release to introduce support for MQ version 9.  Although many Integration Bus use cases can now be achieved without MQ, there is still a very close relationship between the products.  The MQ nodes can be used in message flows to interact with local queue managers via server bindings or remote queue managers using MQ client connections, and in more sophisticated use cases MQ can also be used with IIB to coordinate global transactions, and provide a persistence layer underpinning the Integration Bus aggregation, timer and sequencing nodes.

Since IIBv10, many use cases being tackled by Integration Bus now involve interacting with REST APIs. As such the graphical mapping capability in IIB has been enhanced over the last year or so, to allow users to quickly and easily transform message structures which are described using JSON schema and Swagger documents. With Fix Pack 6, Integration Bus introduces support for these documents to be held inside Application projects in the IIB Toolkit (this is done inside a new “REST API Catalog” folder within the Application) , and also the more user-friendly YAML format is now supported.  Additionally support has been added to allow mapping casts for JSON types, which can help you to work with these kind of structures within the IIB environment tree.

Fix Pack 6 also includes some logging enhancements to the node-wide HTTP listener component. Two new properties have been added to the HTTPConnector named accessLog and accessLogPattern.  You can configure these properties using the mqsichangeproperties command and display their values using the mqsireportproperties command.  If accessLog is set to the value true, then when Integration Bus returns a response to an inbound HTTP request, it will add a new X-IIB-Timing property to the HTTP Header which describes elapsed timings in units of milliseconds for the IIB processing of the request:

• The “read time” is time spent in the servlet part of the IIB HTTP listener.
• The “get wait time” is time spent in the message flow itself.
• The “total time” is a total of read time, get wait time and the time spent in the servlet part of the IIB HTTP Listener after the message flow thread has finished its work.

You can switch on this behaviour using a command like this:
mqsichangeproperties TESTNODE_10006 -b httplistener -n accessLog -v true

You can also provide an accessLogPattern using a command like this:
mqsichangeproperties TESTNODE_10006 -b httplistener -o HTTPConnector -n accessLogPattern -v "%h %l %u %t '%r' %s %b '%{Referer}i' '%{User-Agent}i'  IIB:'%{X-IIB-Timing}o'"

If the accessLogPattern is also configured then an access log file will be generated in the node’s workpath (for example on Windows somewhere like C:\ProgramData\IBM\MQSI\components\TESTNODE_10006\httplistener) whose log entries are described by the accessLogPattern. The syntax for this log pattern matches the Tomcat Access Log Valve syntax.

Fix pack 6 enhances the HTTPInput node’s support for URI Path Segments. Since REST APIs were introduced to IIB as a first-class artifact in version 10, a flow developer has been able to easily use Swagger documents to describe HTTP path parameters. However, for users not wanting to model their REST API using metadata, the HTTPInput node is still frequently used within a conventional message flow.  For this set of users, the URI exposed by the HTTPInput node can include parameters which have an intended meaning for the processing of the message flow. With this new feature, if a user selects the HTTPInput node Advanced property named Parse Query String then in addition to the previously documented behaviour for parsing query strings (such as ?myPram1=my%22Value%22) in to the LocalEnvironment.HTTP.Input.QueryString, there is now an additional new behaviour for populating the LocalEnvironment.HTTP.Input.Path.

For example, consider an HTTPInput node whose Basic properties specify a Path suffix for URL = /MessageFlow1/*

If an HTTP client were to send an HTTP post to http://localhost:7800/MessageFlow1/a%20space/b/c then with this new setting, not only would the request be routed to the message flow deployed with the HTTPInput node in question, the path segments would be placed in to the LocalEnvironment tree as shown below:

HTTP = (
  Input = (
    Path = (
      Segment = 'a space'
      Segment = 'b'
      Segment = 'c'
    )
  )
)

This new capability should make it even easier for developers to interpret the segment paths of URLs when using the HTTPInput node in message flows.

IBM Integration Bus can be configured to publish resource statistics which are collected by an integration node to record performance and operating details of resources used by integration servers. You can view these stats in the IIB web user interface or you can write a program to subscribe to a feed of information on a particular topic.  One type of resource statistics covers Parser Statistics. All message flows create parsers to parse and write input and output messages. Parser statistics can be used to see how much resource is being used by the message trees and bitstreams that these parsers own. For example you can view the approximate amount of user data-related memory is used by a named parser type, and the number of message fields which are associated with a named parser type.  Whilst these statistics are useful to deternine overall parser usage, when performance tuning it can sometimes be difficult to isolate where in a complex flow the parsers are doing all of their work. To assist with this in Fix Pack 6 we have added some new basic activity log entries for parser activity.  To avoid flooding the activity log with very low level information, instead a user can define thresholds which will create warnings when crossed.  Thresholds can be defined for:

• The number of parsers created on a thread.  (Default = 1000)
• The number of fields created on a specific parser. (Default = 100000)

You can configure these thresholds using the mqsichangeproperties command:

• mqsichangeproperties brokerName -e serverName -o ComIbmParserManager -n parserWarningThreshold -v newValue
• mqsichangeproperties brokerName -e serverName -o ComIbmParserManager -n fieldWarningThreshold - v newValue

These settings are incremental thresholds and so:

• When we cross 1000, 2000, 3000 etc parsers on a thread Integration Bus issues a warning in the activity log
• When we cross 100000, 200000, 300000 etc. fields on a specific parser Integration Bus will issue a warning in the activity log.

When carrying out system administration, several mqsi* commands provide an option to reference a file with the extension .broker (using the -n parameter) which contains connection settings for a specific integration node.  Some examples of commands which do this include mqsideploy, mqsilist, mqsistartmsgflow, and mqsicreatepolicy. A full list of commands which have this option is in the Knowledge Center. When executing a command which uses a connection file, then the user is asked to type a password interactively:

C:\Program Files\IBM\IIB\10.0.0.6>mqsilist -n C:\BensConnectionFile.broker
BIP8821W: Enter password for user ID

BIP1060I: Connecting to the integration node...
BIP1115I: Connected to the integration node 'TESTNODE_10006'.
-----------------------------------
BIP1286I: Integration server 'default' on integration node 'TESTNODE_10006' is running.
BIP8071I: Successful command completion.

New in Fix Pack 6 is the capability to optionally include within this connection file a password as well. If this is done then you no longer need to specify a password interactively:

C:\Program Files\IBM\IIB\10.0.0.6>mqsilist -n C:\BensConnectionFileContainsPassword.broker
BIP1060I: Connecting to the integration node...
BIP1115I: Connected to the integration node 'TESTNODE_10006'.
-----------------------------------
BIP1286I: Integration server 'default' on integration node 'TESTNODE_10006' is running.
BIP8071I: Successful command completion.

Here is an example .broker file with the password attribute included:

<?xml version="1.0" encoding="utf-8"?>
<IntegrationNodeConnectionParameters
    Version="10.0.0"
    host="localhost"
    listenerPort="4414"
    integrationNodeName="TESTNODE_10006"
    userName="benthomp"
    password="password1234" 
    xmlns="http://www.ibm.com/xmlns/prod/websphere/iib/8/IntegrationNodeConnectionParameters" />

The Integration Bus web user interface is enabled by default when an integration node is created, but administrators can restrict access by running the mqsichangeproperties command in order to enable SSL on the webadmin component. A keystore can also be configured to store certificates in order for IIB to authenticate inbound client communications.  By default IIB selects the first certificate it finds in the relevant keystore. Fix pack 6 formally introduces support for a user to define a key alias for the web admin component which will then be used to select a specific certificate from the keystore which will then be used when authentication connections to the web user interface. Although this property could be set in previous IIB fix pack levels, with fix pack 6 the mqsireportproperties command has also now been enhanced to report on this property. To try out this new capability:

1. Open ikeyman (or equivalent) and create a key store (for example C:\WebAdminKeyStore\keystore.jks) and populate it with more than one certificate (for a quick test you might want to just create three self signed certificates tom, dick and harry)
2. Enable SSL for the webadmin component:  mqsichangeproperties TESTNODE_10006 -b webadmin -o server -n enableSSL -v true
3. Configure the HTTPS connector port:  mqsichangeproperties TESTNODE_10006 -b webadmin -o HTTPSConnector -n port -v 1433
4. Configure IIB to use the key store you just created:  mqsichangeproperties TESTNODE_10006 -b webadmin -o HTTPSConnector -n keystoreFile -v C:\WebAdminKeyStore\keystore.jks
5. Configure the keyAlias property to choose the certificate:  mqsichangeproperties TESTNODE_10006 -b webadmin -o HTTPSConnector -n keyAlias -v dick

Next, open a web browser and go to the web ui at https://localhost:1433 and you should receive a popup from your browser, to which you can choose to add a security exception and view the serial number of the certificate and it should show a specific match to dick, rather than tom or harry, as the keyAlias is being applied to correctly select the right identity.

To view your settings you can also execute the mqsireportproperties command as follows:

mqsireportproperties TESTNODE_10006 -b webadmin -o HTTPSConnector -r