Explore the new features in IBM Integration Bus version 10.0.0.7

Fix pack 7 introduces a new Kafka drawer on the message flow palette which contains a pair of new message flow nodes for interacting with Kafka messaging systems. Apache Kafka is an open source project that provides a messaging service capability, based upon a distributed commit log, which lets you publish and subscribe data to streams of data records (messages). The Kafka protocol is TCP based and provides a fast, scalable and durable method of decoupling applications using messaging. Kafka maintains feeds of messages in categories called topics.

The KafkaConsumer node is a new input node which connects to a Kafka messaging system, receives messages which are published on a particular Kafka topic, and provide input to the message flow. Data content can be parsed using all the standard IIB message parsers making it easy to receive JSON, XML, BLOB or even tagged/delimited data structures.

The KafkaProducer node is a new output node which publishes an output message from a message flow to a specified topic on a Kafka server. You can set properties on the KafkaProducer node to define how it will connect to the Kafka messaging system, and to specify the topic to which messages are sent. You can also dynamically override the Topic name property using a LocalEnvironment property (OutputLocalEnvironment.Destination.Kafka.Output.topicName) which is passed into the KafkaProducer node.

The Kafka nodes have been built for IIB using the Java Apache Kafka client version 0.10.0.1, which makes it easy to interact with the Message Hub service on Bluemix. 

Message Hub provides a cloud Kafka implementation as a service in Bluemix, which is available in both the US-South (Dallas) and EU-GB (London) data centers. In both cases the service provides a 5 node Kafka cluster which is shared amongst tenants. This gives a user who is new to Kafka a very quick way to avoid needing to do an installation of a Kafka server, where they can quickly and easily set up topics for the exchange of Kafka messages. Because the service is on the cloud you will want to encrypt the communications using TLSv1.2 and also authenticate IIB.

Once you’ve created an IBM Message Hub instance with a set of credentials, you can then use the mqsisetdbparms command to configure the credentials that the KafkaProducer and KafkaConsumer nodes will use to authenticate (in the example below, the node’s name is TESTNODE and the integration server’s name is default):

mqsisetdbparms TESTNODE -n kafka::KAFKA::default -u myUsername -p myPassword

On the Security tab of the nodes, set the Security protocol property to be SASL_SSL and set the SSL protocol to be TLSv1.2.

Also check out this recent blog post, Using the new Kafka nodes in IBM Integration Bus.

There is a new Experimental service named Hybrid Connect which has just been made available in IBM Bluemix:


This service integrates with several IBM software products (including IBM Integration Bus, from Fix Pack 7) and allows you to gather and display product information. You can use the Hybrid Connect service to:

• Register on-premise IBM software products with a Bluemix service in the IBM cloud.
• Gather product information and usage data from enabled and configured on-premise software, such as IBM Integration Bus.
• View information about enabled and configured products through a dashboard in Bluemix.
• Gain a greater insight into the workload of the enabled and configured products, by accessing the runtime usage data.

So long as networking is available, you can connect installations of IBM Integration Bus which are running on-premise, running within Docker containers or virtualized environments. The Hybrid Connect service enables you to connect specific integration servers (or all integration servers on a specified integration node) to a service instance on Bluemix, and report startup and usage data. You can then use the dashboard which is part of the service instance to display the information. In the case of IIB:

• Startup data includes the product install location, product version, APARs applied, and provides a link to the IIB Web Administration console.
• Usage data currently reports the number of active CPUs for the integration server process, and the CPU time being used by the integration server.

An example of the dashboard with seven integration servers registered (3 running in a Docker container and 4 running on a Windows laptop) is shown below:

In order to configure a node named TESTNODE, with an integration server named default, to send usage messages (every 10 minutes) use a command like this:

mqsichangebluemixreporting TESTNODE -e default -c active -b https://hybridconnect-api.ng.bluemix.net/api -r startup -u usage -i 10

You will need to restart TESTNODE for your settings to take effect.

Detailed information on this feature is available in the Knowledge Center.

Another aspect to IBM’s Hybrid Connect initiative is a new IBM Integration Bus feature, delivered in Fix Pack 7, which lets an IIB installation send log entries to the Logmet service in Bluemix. The Logmet service in Bluemix is an IBM hosted implementation of the ELK stack (or more recently referred to as the “Elastic” stack). The term ELK describes a stack of three open source products – Elasticsearch (a clusterable search server), Logstash (a data pipeline technology used to ingest data) and Kibana (a web browser based dashboard technology). This enables you to centralize the logging data for all your IIB installations, and gain an insight into the operation of your systems through a single Kibana dashboard. IIB provides a sample dashboard, which you can choose to have automatically loaded into your Bluemix space when you switch on the logging capability so you can start monitoring in minutes. This dashboard provides a set of charts and graphs which at a glance let you view:

• The machines in your network which are producing the most IIB log events
• The integration servers in your network which are producing the most IIB log events
• Log events charted against time and categorised by their severity
• Integration servers which have been stopped, started or re-configured

You can choose to send logging information for a particular integration server, or for all integration servers on a specified integration node. The command below shows how you can configure all the integration servers belonging to TESTNODE to send log information into a Bluemix space called dev defined in the bthomps@uk.ibm.com organisation in the US-South Bluemix region. The -s parameter is optional and specifies the interval, in seconds, for sending information to the logging service. The minimum value is 1, the maximum value is 300, and the default value is 60. If the integration server is raising an exceptionally high rate of BIP message events, then the integration server will send logging data more often than the configured interval:

mqsichangebluemixreporting TESTNODE -g -l active -z us-south -o bthomps@uk.ibm.com -p dev -n bthomps@uk.ibm.com -w password123 -s 5 -d

You will need to restart TESTNODE for your settings to take effect.

A version 3 Kibana dashboard for IIB is provided for manual import into Logmet, but by default the automated import (if you include the -d parameter on the mqsichangebluemixreporting command) pushes a Kibana version 4 dashboard for IIB. An example is shown below which includes log events pushed from both a Windows IIB installation and also from an Ubuntu installation running within a Docker container:

Detailed information on this feature is available in the Knowledge Center. This feature is initially designated as an experimental capability. Access to the Logmet service in Bluemix is currently provided free of charge and whilst very robust, there are currently no guarantees provided for data retention (although we expect to retain data for 7 days). This means that currently this feature is not recommended for production use, although we expect this status to change with time. If you would like to give us feedback on this capability and join our Hybrid Connect customer program then we would love for you to get in touch. For additional information and to participate, please contact John A. Rodriguez (johnarod@us.ibm.com).

Fix pack 7 of IBM Integration Bus adds support for the Windows 10 operating system for the first time. Windows 10 Enterprise and Windows 10 Professional have both been added to the Windows IBM Integration Bus System Requirements.

For a long time IBM Integration Bus has provided users the opportunity to gather message flow statistics and accounting data, which can be collected to record performance information over a long archive period of several hours or days, or as a short performance snapshot (with records generated every 20 seconds by default). Statistics can be distributed by writing the information to user trace, publishing JSON data or XML data to either an MQ topic or MQTT topic, or if you’re on z/OS you can specify that the data collected is written to SMF (using type 117 records). Fix Pack 7 introduces a new option to distribute statistics information in a Comma Separated Value output format which is written to a collection of rolling log files:

The ComIbmStatsFileWriter resource manager can be used to configure the location, number and size of these files. The new CSV output format can be combined with the existing outputs, so if you would like you can gather statistics in more than one format at the same time. You turn on statistics collection using the mqsichangeflowstats command. When you use the new output format of csv you also need to run the mqsichangeproperties command in order to configure the Statistics File Writer component. This component has properties which specify the desired path for the output files, the number of output files to use and their size. You can also specify whether you would like to include mean average values for each statistic which is included in the written output.

For example, the command below configures the Statistics File Writer component to use a particular directory. This directory must exist when statistics are enabled, and the integration node must have been granted permission to write to it:
mqsichangeproperties TESTNODE -e default -o ComIbmStatsFileWriter -n filePath -v C:\BenStatsOutputDirectory\

Set the numberOfFiles property to control how many files should be used. When a file is full the writer will move to the next file. Once they are all filled, it will begin to overwrite data in the first file. This circular method ensures that the newest data is retained:
mqsichangeproperties TESTNODE -e default -o ComIbmStatsFileWriter -n numberOfFiles -v 3

Set the sizeOfFile property to control the size in megabytes that each file can expand to before the writer rolls to the next file:
mqsichangeproperties TESTNODE -e default -o ComIbmStatsFileWriter -n sizeOfFile -v 10

The averages property is set to the constant value true or false to control whether mean average values are included in the output:
mqsichangeproperties TESTNODE -e default -o ComIbmStatsFileWriter -n averages -v true

Finally, to turn on statistics to take advantage of the File Writer settings use a command like this:
mqsichangeflowstats TESTNODE -e default -s -j -c active -o csv

IBM Integration Bus provides an Activity Log panel within the web user interface to help users quickly understand what their message flows are doing by providing a high-level overview of how IIB is interacting with external resources. This simple concise set of messages can also be written out to a file by defining an Activity Log Configurable Service. The configurable service specifies the file name to be written and some other basic options as shown below. This screen capture shows such an Activity Log configurable service which has been created using the IIB web user interface:

Before the current Fix Pack 7 release, a restart of your Integration Server was required in order for IIB to notice the creation of such a configurable service. This is no longer needed – just create the configurable service and navigate to your nominated file system directory and you will find that IIB has started recording activity log entries to the file:

One aspect of securing IBM Integration Bus is to turn on administration security so that IIB authenticates users who connect to a runtime node from a remote system. It is possible to define a set of web userids and associated passwords for this purpose. If this has been done, then when you attempt to access the IIB web user interface you are prompted to supply this userid / password combination in order to proceed:

When defining the set of allowed userids, you can also tell IIB to interact with an LDAP server in order to authenticate the web userid and supplied password using definitions within LDAP. Here is an example set of commands you could execute to get this up and running:

• mqsichangeproperties TESTNODE -b webadmin -o server -n ldapAuthenticationUri -v \"ldap://IBM425-R9E9V8K:389/dc=ibmexample,dc=com\"
• mqsisetdbparms TESTNODE -n ldap::IBM425-R9E9V8K -u cn=Manager,dc=ibmexample,dc=com -p password123
• mqsiwebuseradmin TESTNODE -c -u bthomps -x -r iibuser
• mqsistop TESTNODE
• mqsichangeauthmode TESTNODE -s active -m file
• mqsistart TESTNODE

This approach of enabling an integration node to use LDAP for authentication has proven very popular with our users since it was first introduced earlier this year in IIBv10.0.0.4 Let’s focus for a moment on the third mqsiwebuseradmin command, which was shown in bold typeface above. The -x parameter on this command is to instruct IIB, that when a user types in bthomps we should pass this userid, along with the password supplied by the login attempt, through to LDAP. If bthomps is found to have the right password in LDAP, then the user is given the privileges of the local iibuser user which is defined in the operating system of the IIB installation. Here’s a view of a simple LDAP setup where the bthomps userid is defined to carry the password of value password1:

Fix Pack 7 carries a further enhancement in this area. Until now, even with a centralised LDAP store for the desired set of users, an IIB administrator would need to invoke the mqsiwebuseradmin command for every web user they wished to define to the IIB system. A new wildcard syntax has been introduced in Fix Pack 7 which means that ALL userids can be passed through to LDAP:

• mqsiwebuseradmin TESTNODE -c -u '*' -x -r iibuser

So, if the IIB administrator had issued the above mqsiwebuseradmin command instead, then all of the users (bthomps, mboag, rconver, snagcho and tdolby) could be used and authenticated without the need for any further IIB commands to be executed.

When using IIB message flows for HTTP based scenarios, it is common practice to implement SSL authentication for the IIB runtime which involves setting up a public key infrastructure (PKI). Examples of message flow nodes which take advantage of such configuration include the HTTPInput node, the SOAPInput node, the HTTPRequest node, the SOAPRequest node and the RESTRequest node. In defining a PKI, a user will typically create a keystore file and / or a truststore file and then either create or import certificates which allow the authentication of the participants in the scenario:

• A keystore holds the private keys and public key certificates for an application.
• A truststore contains the CA certificates required to authenticate certificates that are presented by another application.

Background information on Digital Certificates can be found here. Until Fix Pack 7, on both distributed platforms and also on z/OS, IBM Integration Bus supported the use of JKS keystores. Another common alternative, specifically on the z/OS platform, is to use JCE RACF keystores. From Fix Pack 7, IIB introduces support to allow the storage of keys and certificates in RACF key rings (a key ring is a collection of certificates that identify a networking trust relationship). These RACF keystores can be used in several different areas of IIB configuration:

• Integration node level (one keystore, and one truststore for each integration node).
• Integration node listener level (one keystore and one truststore for the integration node listener in an integration node).Integration node listeners that do not have a PKI configured use the integration node level PKI configuration.
• Integration server level (one keystore and one truststore for each integration server). Integration servers that do not have a PKI configured use the integration node level PKI configuration.
• Embedded listener level (one keystore and one truststore for the embedded listener in an integration server). Embedded listeners that do not have a PKI configured use the integration server level PKI configuration. If there is no integration server level PKI configuration, then the embedded listener uses the integration node level PKI configuration.
• Webadmin HTTPSConnector level

Although not strictly part of the Fix Pack 7 release, just to round off today’s post, a quick mention whilst we have your attention that the process of running IBM Integration Bus inside Docker as part of the Bluemix Container service has got dramatically simpler recently with the provision of a pre-built image containing IIB Developer Edition.