Explore the new features in IBM Integration Bus version 10.0.0.9

This fix pack extends the facility which reports IIB logging information to a centralised cloud-hosted IBM Bluemix dashboard in the Logmet service, to also provide the reporting of statistics and accounting data. For example, consider an IIB node named TESTNODE_10009, with an integration server called default. The mqsichangebluemixreporting command can be used to make logging active, and then the mqsichangeflowstats command can be used to configure the integration server to also emit statistics to Bluemix, using the new value of bluemix for the -o parameter, as shown below:

• mqsichangebluemixreporting TESTNODE_10009 -l active -e default -r us-south -o ExampleUserid@uk.ibm.com -a IIBspace -u ExampleUserid@uk.ibm.com -p Password123
• mqsistop TESTNODE_10009
• mqsistart TESTNODE_10009
• mqsichangeflowstats TESTNODE_10009 -g -s -j -o bluemix -c active
• mqsireportflowstats TESTNODE_10009 -e default -j -s

The screenshot below shows a simple visualization of a few selected values from flow statistics including, for each 20 second snapshot interval:

• The Total Number of Input Messages
• The Average Input Size
• The Average Elapsed Time
• The Average CPU Time

Detailed information on this feature is available in the Knowledge Center.

IBM Integration Bus v10.0.0.7 and v10.0.0.8, provided integration features for the Product Insights service on Bluemix to gather and display rutime product information and usage metrics.Building on these capabilities, fix pack 9 adds support for using the Product Insights Secure Gateway. This optional Gateway component (based upon DataPower technology) lets you view the data sent in all interactions between our software products and the Product Insights service. It provides security assurances and the ability to audit. As you can see in the following illustration, the gateway sits in the network between IBM software products (in this example IBM Integration Bus) and the Product Insights service. All data flowing through the gateway is logged, and you can view those logs from a gateway user interface.

Typically the Gateway component will be deployed in a DMZ and the communications from IIB to the Gateway will be transmitted via HTTPS, so to configure this feature you must set up IIB to trust the public certificate of the gateway by importing it into the integration server’s truststore. You can do this using keytool and then use the mqsichangeproperties command to configure IIB’s truststore. The IIB Knowledge Center describes the required steps in detail, and this Product Insights page describes configuration of the Gateway itself, which is normally configured though a web user interface.

Typically, IBM Integration Bus is deployed within a company’s internal network, close to Systems of Record. This means that in some situations IIB may be positioned behind a firewall with restrictions on the direct network connectivity which is permitted when sending data to SaaS applications or internet services. Many companies will deploy an HTTP Forward Proxy server which sits between applications such as IIB and the outbound internet connection. The proxy server will intermediate on behalf of several applications, acting as a single point of control for the traffic, keeping a record of the requests which have been passed through and ensuring that responses are routed back to the right application. The diagram below shows that connections from multiple integration servers can be coordinated through a single HTTP proxy. Fix pack 9 extends IIB’s capabilities to interact with an HTTP forward proxy in several areas, so this diagram serves as a good overview picture for the current section of this blog post discussing Bluemix Product Insights, as well as the later sections discussing other nodes such as the Salesforce Request, HTTPRequest node, SOAPRequest and RESTRequest.

You can now configure integration servers to connect to the Product Insights service through an HTTP proxy server. You can do this by specifying the -y httpProxy parameter on the mqsichangebluemixreporting command, for example as shown below:

• mqsichangebluemixreporting TESTNODE_10009 -g -c active -j https://hybridconnect-api.ng.bluemix.net -k 9ec5g5c2-5cm1-3287-2bxd-3c71fc22d467 -y https://myHTTPproxyaddress.hursley.ibm.com

To remove the setting for a previously specified proxy, set the -y parameter to “”. For example like this:

• mqsichangebluemixreporting TESTNODE_10009 -g -c active -j https://hybridconnect-api.ng.bluemix.net -k 9ec5g5c2-5cm1-3287-2bxd-3c71fc22d467 -y ""

Typically, an HTTP Proxy Server will require a connecting application to authenticate itself through the provision of a username and password. You can provide IIB with these credentials using the mqsisetdbparms command with the new resource name httpproxy. The address of the gateway which was specified on the mqsichangebluemixreporting command is used as a key to locate the correct username / password combination. So, extending the example used in the above commands, you would provide the username and password like this:

• mqsisetdbparms TESTNODE_10009 -n httpproxy::myHTTPproxyaddress.hursley.ibm.com -u myProxyUserID -p myProxyPassword

IIBv10.0.0.9 adds a new optional parameter to the Salesforce Request node named HTTP(S) proxy location which lets the flow developer specify where their HTTP forward proxy server lives in the network using a value in the form hostname:port. The new property is shown on the node’s properties in the screenshot below:

If the HTTP proxy server does not have authentication enabled, then this parameter is all that is needed. If the HTTP proxy server does have authentication enabled, then the username and password should be specified using the mqsisetdbparms command like this:

• mqsisetdbparms TESTNODE_10009 -n httpproxy::myHTTPproxyaddress.hursley.ibm.com -u myProxyUserID -p myProxyPassword

If the proxy server name on the node matches with a proxy server specified using the mqsisetdbparms command then IIB will use these credentials when connecting to the proxy.

The last two sections discussed how to link together the credentials for an HTTP Proxy Server which are provided as part of the mqsisetdbparms configuration command, with Product Insights and the Salesforce Request node respectively, but it is also possible to specify a default set of credentials to be used in situations where there has been no match for a given HTTP proxy server. The command to specify a default set of credentials is as follows:

• mqsisetdbparms TESTNODE_10009 -n httpproxy::HTTPPROXY -u myDefaultProxyUserID -p myDefaultProxyPassword

This default capability works with several areas of the product which have existing support for the use of an HTTP forward proxy:

• HTTPRequest node
• HTTPAsyncRequest node
• SOAPRequest node
• SOAPAsyncRequest node
• RESTRequest node
• RESTAsyncRequest node

The Trace node primitive lets IIB developers record the contents of a message as it passes through a message flow. This helps you to check on the actions which a specific node in a flow has had on the message as it passes. Although trace nodes can be turned on and off, you must insert one in to a flow at the relevant point (which is a design-time activity) for this technique to be effective. There may be circumstances where you would like to check on the message tree as it passes through a message flow which you have already deployed. With this requirement in mind, fix pack 9 adds a new servicability feature which provides a new trace level named debugTree which can be applied using the mqsichangetrace command. Using this trace level will cause tracing to record the message tree structure as it traverses along each wired branch of the flow. For example, consider an example message flow such as the Transformation_ESQL flow which is provided in the tutorial gallery for showing “Transformation using ESQL in a message flow” and which consists of HTTPInput, Compute and HTTPReply. You can turn on debugTree tracing like this:

• mqsichangetrace TESTNODE_10009 -e default -u -l debugTree

If you then inject the provided test message, you can gather user tracing in the normal way like this:

• mqsireadlog TESTNODE_10009 -e default -u -o C:\trace.trc
• mqsiformatlog -i C:\trace.trc -o C:\trace.txt

The resulting trace file, trace.txt, records the message tree structure which is propagated between the nodes in the flow, as shown in the extract below:

2017-05-31 12:49:29.563212    13844   UserTrace   BIP2299I: Message propagated from node ''HTTP Input'' to node ''Compute'' in message flow ''Transformation_ESQL''. The state of message trees are recorded as ''
                                       
Message : 
                                       
( ['WSRoot' : 0x21a29310]
  (0x01000000:Name  ):Properties      = ( ['WSPROPERTYPARSER' : 0x6d27080]
   (0x03000000:NameValue):MessageSet             = '' (CHARACTER)
   (0x03000000:NameValue):MessageType            = '' (CHARACTER)
   (0x03000000:NameValue):MessageFormat          = '' (CHARACTER)
   (0x03000000:NameValue):Encoding               = 546 (INTEGER)
   (0x03000000:NameValue):CodedCharSetId         = 1208 (INTEGER)
   (0x03000000:NameValue):Transactional          = FALSE (BOOLEAN)
   (0x03000000:NameValue):Persistence            = FALSE (BOOLEAN)
   (0x03000000:NameValue):CreationTime           = GMTTIMESTAMP '2017-05-31 11:49:29.561001' (GMTTIMESTAMP)
   (0x03000000:NameValue):ExpirationTime         = -1 (INTEGER)
   (0x03000000:NameValue):Priority               = 0 (INTEGER)
   (0x03000000:NameValue):ReplyIdentifier        = X'000000000000000000000000000000000000000000000000' (BLOB)
   (0x03000000:NameValue):ReplyProtocol          = 'SOAP-HTTP' (CHARACTER)
   (0x03000000:NameValue):Topic                  = NULL
   (0x03000000:NameValue):ContentType            = 'application/x-www-form-urlencoded' (CHARACTER)
   (0x03000000:NameValue):IdentitySourceType     = '' (CHARACTER)
   (0x03000000:NameValue):IdentitySourceToken    = '' (CHARACTER)
   (0x03000000:NameValue):IdentitySourcePassword = '' (CHARACTER)
   (0x03000000:NameValue):IdentitySourceIssuedBy = '' (CHARACTER)
   (0x03000000:NameValue):IdentityMappedType     = '' (CHARACTER)
   (0x03000000:NameValue):IdentityMappedToken    = '' (CHARACTER)
   (0x03000000:NameValue):IdentityMappedPassword = '' (CHARACTER)
   (0x03000000:NameValue):IdentityMappedIssuedBy = '' (CHARACTER)
  )
  (0x01000000:Name  ):HTTPInputHeader = ( ['WSINPHDR' : 0x21a29ca0]
   (0x03000000:NameValue):X-Server-Name           = 'localhost' (CHARACTER)
   (0x03000000:NameValue):X-Server-Port           = '7800' (CHARACTER)
   (0x03000000:NameValue):Host                    = 'localhost:7800' (CHARACTER)
   (0x03000000:NameValue):Content-Length          = '1531' (CHARACTER)
   (0x03000000:NameValue):X-Original-HTTP-Command = 'POST http://localhost:7800/Transformation_ESQL HTTP/1.1' (CHARACTER)
   (0x03000000:NameValue):X-Remote-Addr           = '127.0.0.1' (CHARACTER)
   (0x03000000:NameValue):User-Agent              = 'Java/1.7.0' (CHARACTER)
   (0x03000000:NameValue):X-Scheme                = 'http' (CHARACTER)
   (0x03000000:NameValue):X-Remote-Host           = '127.0.0.1' (CHARACTER)
   (0x03000000:NameValue):Content-Type            = 'application/x-www-form-urlencoded' (CHARACTER)
   (0x03000000:NameValue):Connection              = 'keep-alive' (CHARACTER)
   (0x03000000:NameValue):Accept                  = 'text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2' (CHARACTER)
  )
  (0x01000000:Folder):XMLNSC          = ( ['xmlnsc' : 0x20d50470]
   (0x01000400:XmlDeclaration):XmlDeclaration = (
    (0x03000100:Attribute):Version  = '1.0' (CHARACTER)
    (0x03000100:Attribute):Encoding = 'UTF-8' (CHARACTER)
   )
   (0x01000000:Folder        ):SaleEnvelope   = (
    (0x01000000:Folder):Header   = (
     (0x03000000:PCDataField):SaleListCount      = '1' (CHARACTER)
     (0x03000000:PCDataField):TransformationType = 'compute' (CHARACTER)
    )
    (0x01000000:Folder):SaleList = (
     (0x01000000:Folder):Invoice = (
      (0x03000000:PCDataField):Initial  = 'K' (CHARACTER)
      (0x03000000:PCDataField):Initial  = 'A' (CHARACTER)
      (0x03000000:PCDataField):Surname  = 'Braithwaite' (CHARACTER)
      (0x01000000:Folder     ):Item     = (
       (0x03000000:PCDataField):Code        = '00' (CHARACTER)
       (0x03000000:PCDataField):Code        = '01' (CHARACTER)
       (0x03000000:PCDataField):Code        = '02' (CHARACTER)
       (0x03000000:PCDataField):Description = 'Twister' (CHARACTER)
       (0x03000000:PCDataField):Category    = 'Games' (CHARACTER)
       (0x03000000:PCDataField):Price       = '00.30' (CHARACTER)
       (0x03000000:PCDataField):Quantity    = '01' (CHARACTER)
      )
      (0x01000000:Folder     ):Item     = (
       (0x03000000:PCDataField):Code        = '02' (CHARACTER)
       (0x03000000:PCDataField):Code        = '03' (CHARACTER)
       (0x03000000:PCDataField):Code        = '01' (CHARACTER)
       (0x03000000:PCDataField):Description = 'The Times Newspaper' (CHARACTER)
       (0x03000000:PCDataField):Category    = 'Books and Media' (CHARACTER)
       (0x03000000:PCDataField):Price       = '00.20' (CHARACTER)
       (0x03000000:PCDataField):Quantity    = '01' (CHARACTER)
      )
      (0x03000000:PCDataField):Balance  = '00.50' (CHARACTER)
      (0x03000000:PCDataField):Currency = 'Sterling' (CHARACTER)
     )
     (0x01000000:Folder):Invoice = (
      (0x03000000:PCDataField):Initial  = 'T' (CHARACTER)
      (0x03000000:PCDataField):Initial  = 'J' (CHARACTER)
      (0x03000000:PCDataField):Surname  = 'Dunnwin' (CHARACTER)
      (0x01000000:Folder     ):Item     = (
       (0x03000000:PCDataField):Code        = '04' (CHARACTER)
       (0x03000000:PCDataField):Code        = '05' (CHARACTER)
       (0x03000000:PCDataField):Code        = '01' (CHARACTER)
       (0x03000000:PCDataField):Description = 'The Origin of Species' (CHARACTER)
       (0x03000000:PCDataField):Category    = 'Books and Media' (CHARACTER)
       (0x03000000:PCDataField):Price       = '22.34' (CHARACTER)
       (0x03000000:PCDataField):Quantity    = '02' (CHARACTER)
      )
      (0x01000000:Folder     ):Item     = (
       (0x03000000:PCDataField):Code        = '06' (CHARACTER)
       (0x03000000:PCDataField):Code        = '07' (CHARACTER)
       (0x03000000:PCDataField):Code        = '01' (CHARACTER)
       (0x03000000:PCDataField):Description = 'Microscope' (CHARACTER)
       (0x03000000:PCDataField):Category    = 'Miscellaneous' (CHARACTER)
       (0x03000000:PCDataField):Price       = '36.20' (CHARACTER)
       (0x03000000:PCDataField):Quantity    = '01' (CHARACTER)
      )
      (0x03000000:PCDataField):Balance  = '81.84' (CHARACTER)
      (0x03000000:PCDataField):Currency = 'Euros' (CHARACTER)
     )
    )
    (0x01000000:Folder):Trailer  = (
     (0x03000000:PCDataField):CompletionTime = '12.00.00' (CHARACTER)
    )
   )
  )
 )
                                       
LocalEnvironment : 
                                       
 ( ['MQROOT' : 0x21a28650]
  (0x01000000:Name):Destination = (
   (0x01000000:Name):HTTP = (
    (0x03000000:NameValue):RequestIdentifier = X'48545450000000000000000059db565e1436000000000000' (BLOB)
   )
  )
 )
                                       
Environment :                                     
                                                                              
ExceptionList : 
                                       
  ''. 
  A message is propagated from node ''HTTP Input'' to node ''Compute'' in message flow ''Transformation_ESQL''. The Message, LocalEnvironment, Environment and ExceptionList trees are logged as above. 
  This message is for information only.

For those with an eye for detail, when executing the mqsireportproperties command against the ImbESQLManager in IIBv10.0.0.9 you may notice a couple of new properties which are reported named performanceStatsReportingOn and performanceStatsOutputPath. For a new integration server you should see a report like this:

mqsireportproperties TESTNODE_10009 -e default -o ImbESQLManager -r

ImbESQLManager
  uuid='ImbESQLManager'
  userTraceLevel='none'
  traceLevel='none'
  userTraceFilter='none'
  traceFilter='none'
  castWithAccurateZoneOffsets='no'
  alwaysCastToLocal='yes'
  useICUStringManipulation='no'
  allowPropagateWithSharedLockHeld='no'
  performanceStatsReportingOn='inactive'
  performanceStatsOutputPath=''

BIP8071I: Successful command completion.

You can enable performance statistics reporting for ESQL using a command like this:

• mqsichangeproperties TESTNODE_10009 -e default -o ImbESQLManager -n performanceStatsReportingOn,performanceStatsOutputPath -v active,C:\ESQLStats

The statistics should be written to a file named <IntegrationNodeName>_<IntegrationServerName>_ESQLPerfStats.txt in the nominated output directory. The contents of the file show you how your ESQL code has been performing, line by line:

       GMT Timestamp        TID  [  uSec  ] [  Cycles  ] [      Address     ] RDL: Statements...
2017-05-31 19:49:00.458927 14232 [    4671] [  10225564] [0x0000000001B7F3C0] RDL: Begin Compute Node Transformation_ESQL.Compute
2017-05-31 19:49:00.458949 14232 [    4646] [  10176920] [0x000000000C246890] RDL:  SCHEMA "" PATH ""
2017-05-31 19:49:00.458953 14232 [    4641] [  10168472] [0x000000000C251000] RDL:   MODULE Transformation_Compute
2017-05-31 19:49:00.458957 14232 [    4635] [  10156572] [0x000000000C246260] RDL:    FUNCTION Main(...)
2017-05-31 19:49:00.459034 14232 [    4550] [   9967220] [0x000000000C195B00] RDL:     BEGIN
2017-05-31 19:49:00.459038 14232 [     863] [   1891332] [0x000000000BC59040] RDL:      CALL CopyMessageHeaders();
2017-05-31 19:49:00.459084 14232 [     815] [   1786668] [0x000000000C246470] RDL:      PROCEDURE CopyMessageHeaders(...)
2017-05-31 19:49:00.459115 14232 [     781] [   1712468] [0x000000000C2D6510] RDL:       BEGIN
2017-05-31 19:49:00.459117 14232 [      22] [     49388] [0x000000000BD6FAC0] RDL:        DECLARE I INTEGER 1;
2017-05-31 19:49:00.459140 14232 [      19] [     40920] [0x000000000BD6FC30] RDL:        DECLARE J INTEGER;
2017-05-31 19:49:00.459161 14232 [      76] [    164696] [0x000000000BC59BF0] RDL:        SET J = CARDINALITY(InputRoot.*:*[]);
2017-05-31 19:49:00.459239 14232 [     656] [   1439504] [0x000000000C2EF400] RDL:        WHILE I < J DO
2017-05-31 19:49:00.459297 14232 [     244] [    533724] [0x000000000C2EF4A0] RDL:         Begin
2017-05-31 19:49:00.459301 14232 [     182] [    401744] [0x000000000BC59CA0] RDL:          SET OutputRoot.*:*[I] = InputRoot.*:*[I];
2017-05-31 19:49:00.459488 14232 [      52] [    114100] [0x000000000BC59D50] RDL:          SET I = I + 1;
2017-05-31 19:49:00.459541 14232 [     244] [    533724] [0x000000000C2EF4A0] RDL:         End;
2017-05-31 19:49:00.459580 14232 [     259] [    565316] [0x000000000C2EF4A0] RDL:         Begin
2017-05-31 19:49:00.459583 14232 [     186] [    408928] [0x000000000BC59CA0] RDL:          SET OutputRoot.*:*[I] = InputRoot.*:*[I];
2017-05-31 19:49:00.459773 14232 [      65] [    142168] [0x000000000BC59D50] RDL:          SET I = I + 1;
2017-05-31 19:49:00.459839 14232 [     259] [    565316] [0x000000000C2EF4A0] RDL:         End;
2017-05-31 19:49:00.459895 14232 [     656] [   1439504] [0x000000000C2EF400] RDL:        END WHILE;
2017-05-31 19:49:00.459896 14232 [     781] [   1712468] [0x000000000C2D6510] RDL:       END;
2017-05-31 19:49:00.459899 14232 [     815] [   1786668] [0x000000000C246470] RDL:      END Procedure CopyMessageHeaders;

The FileOutput node has been enhanced in IIBv10.0.0.9 to enable a flow developer to override the permissions used when creating an output file. Consider a simple example message flow with a FileOutput node as shown in the screen shot below:

The default permission given to the file created by the FileOutput node is enforced by the Operating System where IIB is running based upon the permissions of the parent folder and the umask of the integration server (DataFlowEngine) process. You can change the permissions used by setting a LocalEnvironment override to specify a permissions string using a 3 digit octall string (the same format as the unix chmod command). For example, the ESQL shown below demonstrates how to set the override:

CALL CopyMessageHeaders();
SET OutputRoot.BLOB.BLOB = CAST('Hello World' AS BLOB CCSID 1208);
SET OutputLocalEnvironment = InputLocalEnvironment;
SET OutputLocalEvironment.Destination.File.PosixPermissions = '555';

As shown in the output from the listing command below, you can see that the output.txt file has been generated with the correct r-xr-xr-x permissions:

root@osboxes:/home/osboxes/Desktop/output# ls -la
total 16
drwxrwxrwx 3 root    root    4096 May 31 22:44 .
drwxr-xr-x 4 osboxes osboxes 4096 May 31 22:33 ..
drwxrwx--x 2 root    root    4096 May 31 22:44 mqsitransit
-r-xr-xr-x 1 root    root      11 May 31 22:44 output.txt
root@osboxes:/home/osboxes/Desktop/output# more output.txt
Hello World
root@osboxes:/home/osboxes/Desktop/output#

We have also published a further 4 new tutorials in the IIB Toolkit Tutorials Gallery on these topics:

• Using a LoopBack Request node to insert data into a Cloudant database
• Using a Workload Management Policy to restrict the maximum rate at which a message flow can run
• Using a Sequence node to inject a sequence number into a message
• Using a Resequence node to reorder messages going through a flow

As we continue to enhance this area of the product, if there are particular topics which you would like us to cover in future tutorials then please let us know!