IBM Integration Bus Advanced Logging Configuration

syslog-ng

This allows fairly complex configuration of the messages. You need to do some extra work to configure it, but the results tend to be great.

//get a hook into the raw syslog data stream
source src { internal(); unix-dgram("/dev/log"); }; 
//set up a filter to extract only the desired messages
filter f_iiblog     { match("IBM Integration Bus") and match("BIP"); }; 
//define where you want to log messages
destination iiblog { file("/home/mqm/iiblog/iib.log" owner(mqm) group(mqbrkrs) perm(0644)); }; 
//put it all together to get the output
log { source(src); filter(f_iiblog); destination(iiblog); };

rsyslog

This program is very complex with a ton of functionality. It can probably do more than you ever want with your syslog. Order matters in this configuration, so be careful how you order your statements. For IIB, you would need something like:

//set the permissions on the file to be created
$fileCreateMode 0644 
//write the IIB messages to a log
:msg, contains, "IBM Integration Bus" /var/log/iib.log
//remove all IIB messages from subsequent logs generated later in the config file
:msg, contains, "IBM Integration Bus" ~
//write all non-IIB message to a different log
*.* /var/log/syslog

This program is great for complex logging. For example, you can have it send the log entries over an encrypted channel to another machine if you desire. And it even has some built in log rotation functions.

Event Viewer

The provided event viewer can be configured to generate better results.

Open the event viewer. In the left hand pane, expand Windows Logs and select the Application log. In the right hand pane (not the file menu) is the Actions section.

The first action you will likely want to make is to increase the log size limit and/or enable archiving and rotation. To do this, choose the Properties action. In the properties, you should see the default log size of 20MB. Increase that size as needed. Below are the options for archiving/rotation.

Once your log can hold more than a few events, create a custom view to extract the desired messages. Use the Create Custom View action. This will act more like a permanent filter that will appear in the left hand pane under Custom Views. To log all IIB messages, select Logged = Anytime, Event level = All, select the “By source” radio button, and in the drop down list, select the IIB version and fix pack. Here you will see there is an entry for each installed fix pack level. Thus if you install a new fix pack, you will need to update your view later.

You can add more filters if desired, such as user id or specific event messages.

Once you have a custom view, you have a couple more options that may be of use. Open your custom view by selecting it in the left pane. In the right pane are two actions you can set. You can attach a task for specific events or to the entire view. You could for instance have it send an email whenever a integration node (broker) or integration server (EG) starts so that you can monitor for unexpected abends. Or you could start a script to take an action on a specific event (restart the integration node automatically, clear disk space, etc.).

Syslog-ng and rsyslog both allow for splitting and filtering as demonstrated before. Using the same approach, you can extend the sample to allow greater granularity in your logs. For instance, you could have a log for each integration node or each integration server. All you need to know is the command and the syntax used by the input. The inputs are in this format, where {} denotes a variable:

IBM Integration Bus v{####}[{#*, optional}]: ({integration server name}.{integration node name, if applicable})[{Thread}] (Msg {#}/{#}{optional})BIP####{I, W, E}: {message}

Examples:
IBM Integration Bus v9001[4718858]: (NODE)[1]BIP8873I: Starting the component verification for component ‘SERVER’.
IBM Integration Bus v9003 (NODE.SERVER) [Thread 5743] (Msg 1/6) BIP2628E: Exception condition detected on input node ‘MyFlow.MQInput Node’.

To filter by integration server, you would search for the integration server’s name in the entry:
Syslog-ng:

filter f_nodelog { match("IBM Integration Bus") and match ("NODE"); }; 
rsyslog: :msg, contains, "NODE" /var/log/server.log

Windows doesn’t have a true splitting feature. All entries are routed to the same location, then filters are applied on top. By saving the filter to the daemon, you get a pseudo-split as each filter is split, but you rerun the filter on each invocation. To do this, it helps to select an event in the viewer first, then select the ‘Details’ tab followed by the ‘XML view’. Next, create a filter like mentioned above, stopping before clicking the ‘OK’ button. Once ready, switch to the ‘XML’ tab. This will be populated with the details of the ‘Filter’ tab. At the bottom is a check box to ‘Edit query manually’. Check this and accept the warning if it pops up. If you have the event pulled up in the background, it should look something like this:

Your event may have more information in the ‘EventData’ depending on the message output. The only field that will be there for sure is the integration node name.

To complete the filter, you need to extract the integration node name, called ‘node_w_q’ in this image. In the query, the ‘Select’ element is an XPath 1.0 statement, which has some limitations, but is usable for our needs. Before the end tag for the ‘Select’, you now need to add our additional filter options. The filter you need to add looks like:

and *[EventData[Data and (Data=’node_w_q’)]]

The final query looks like this:

<QueryList>
   <Query Id="0" Path="Application">
     <Select Path="Application">*[System[Provider[@Name='IBM Integration v10006']]] and *[EventData[Data and (Data='node_w_q')]]</Select>
   </Query>
</QueryList>

Here it is important to note that the event is generated using both the integration node and integration server name in one field if applicable. Thus the above query would only pull out messages for the integration node, but not the integration servers within that integration node. To get the integration servers, you would need a different filter as there is no wildcard support nor are string functions (such as contains) supported:

<QueryList>
  <Query Id="0" Path="Application">
    <Select Path="Application">*[System[Provider[@Name='IBM Integration v10006']]] and (*[EventData[Data and (Data='node_w_q.default')]] or *[EventData[Data and (Data='node_w_q')]]) </Select>
  </Query>
</QueryList>

In this example, the integration server is called ‘default’. You would need to update the query every time you add a new integration server to your integration node.

More information:
Consuming Events

Built-in to the OS you should find the logrotate command: https://linux.die.net/man/8/logrotate

This command does rotation and archiving in one step (if desired). The sample rotation would look like:

/var/mqsi/components/<integration node>/<integration server>/stderr {
size 100m
compress
missingok
create 644 mqbrkrs mqbrkrs
}

This will compress (compress) the log when it reaches 100MB (size 100m), using permissions 644 and mqbrkrs owner/group, and don’t throw an error if the log is currently missing (missingok).

The command has other functionality you can use. For example, keep only X number of archives, rotating based on time instead of size, emailing archives, or moving the archives to a different directory.

Event Viewer

The event viewer by default overwrites itself when the log reaches 20MB. You can change this limit as per above. In the same location you can enable the archiving.

The trace node is the best logging tool available to you. These nodes will not affect your message trees, they are dynamic, and you can turn all the nodes on/off from the command, so that they are only enabled when you need to investigate an issue. You can have it write to traces, files, or the syslog. For instance, the following pattern will output all the message trees and will uniquely identify the exact integration node, integration server, application, flow, and node that wrote the message:

============================== S T A R T   O F   T R A C E =============================
                  Time:     ${CURRENT_TIMESTAMP}
   IntegrationNodeName:     ${BrokerName}  
      QueueManagerName:     ${QueueManagerName}
IntegrationNodeVersion:     ${BrokerVersion}  
              Platform:     ${Family}
             ProcessID:     ${ProcessId} 
 IntegrationNodeUserId:     ${BrokerUserId}
IntegrationServerLabel:     ${ExecutionGroupLabel}
      ApplicationLabel:     ${ApplicationLabel}
      MessageFlowLabel:     ${MessageFlowLabel}
             NodeLabel:     ${NodeLabel}
==============================     E X C E P T I O N S     =============================
${ExceptionList}     
========================   L O C A L    E N V I R O N M E N T   ========================
${LocalEnvironment}      
==============================    E N V I R O N M E N T    =============================
${Environment}    
==============================          R O O T         ================================
${Root}
==============================   E N D   O F   T R A C E   =============================

You can find a list of all properties that are available here:
Integration node properties that are accessible from ESQL, the Mapping node, and Java

Placing a trace node on every failure terminal and catch terminal will make sure that all the exceptions are logged along with where in the flow the issue occurred and what the message being processed was. Use the node between nodes in the flow to make sure that the data leaving the prior node and entering the next node is well formed and has the expected data.

Trace nodes can be enabled or disabled by using this command:
mqsichangetrace

Caveat: When you select the user trace or local system log as output destinations, you may find that data is truncated. For instance, the user trace usually only logs 16KB of data. On Linux/Unix, the syslog entry is usually only one line long. Using the direct file write option is the best option if you have lots of data per entry to log.

More information:
Trace node

Throw nodes are used primarily to throw exceptions, which cause a roll back and failure or catch processing to be executed. In addition, when the error is thrown, you can choose the error number and error message. This can be used to log a more user friendly error to the log and/or the client.

Throw nodes work best if preceded by a trace node. If you connect a trace node to each failure terminal, then all exceptions will be caught, however, if there are no subsequent nodes, processing will end and the transaction will commit. You would want to use the throw node after the trace to invoke your error or catch terminal processing. For instance, this simple flow receives a SOAP input, makes a SOAP request, and then replies. When thorough logging is added, it looks more complicated, but the additions will help identify all problems that could occur and will auto inform the correct team members to investigate the issues, as the ideal flow would not have errors:

The four failures that are re-thrown at the top roll back to the SOAP Input node where the catch processing is then invoked.

More information:
Throw node

These commands allow you to throw dynamic errors and record log entries at the exact moment they occurred within your code rather than letting a faulty message pass to other nodes. They work much in the same manner as the Throw node, but they are more dynamic. The Throw node will always output the same static message you tell it to. With the throw commands, you can change that message as you need. You can thus make the errors personal or contain additional debugging information not available to other nodes. For example, you could add database query results that affected the outcome.

More information:
THROW statement
JavaCompute node exception handling

This command is a bit different than we’ve discussed thus far. The above all either write log entries and/or throw errors. This command does neither. Instead, it is used as part of the error handling process. The throw commands will stop the current processing and cause a roll back when executed. However, you may not want to stop processing, just perform the logging or roll back only so far. For example, if you alter the above flow, you could remove the SOAPRequest fault and failure processing. To handle those errors, you could have added a DECLARE HANDLER in the Transform node. The code could have altered the SOAPRequest configurable properties to point to a different backend, then reissue the request to the new target. If that failed, it could have then used the THROW statement to throw a different error, such as “Attempted all backend servers and did not get a successful response. See the following errors from each server attempted”. This would make the flow more robust while simplifying the logging process.

Additional information:
DECLARE HANDLER statement