App Connect

From ACE 12.0.12.0 the Toolkit palette has been expanded to include 4 new Discovery Input Message Flow nodes:

• GitLab Input node: Use the GitLab Input node to trigger a message flow and receive input from GitLab. For example, you can use the GitLab Input node to monitor GitLab for new issues.
• LDAP Input node: Use the LDAP Input node to trigger a message flow and receive input from LDAP. For example, you can use the LDAP Input node to monitor LDAP for changes to devices or entries representing names or personnel.
• Microsoft Entra ID Input node: Use the Microsoft Entra ID Input node to trigger a message flow and receive input from Microsoft Entra ID. For example, you can use the Microsoft Entra ID Input node to monitor Microsoft Entra ID for new users.
• Salesforce Commerce Cloud Digital Data Input node: Use the Salesforce Commerce Cloud Digital Data Input node to trigger a message flow and receive input from Salesforce Commerce Cloud Digital Data. For example, you can use the Salesforce Commerce Cloud Digital Data Input node to monitor Salesforce Commerce Cloud Digital Data for new customers.

For the last several mod releases, ACE software has been expanded with large numbers of new Discovery Connector message flow nodes which are typically used when integrating with third party SaaS applications. When deploying message flows containing Discovery Connector nodes within an on premises data center, for some users it might be considered necessary to go through an HTTP proxy when connecting to such public endpoints. With this requirement in mind, ACE 12.0.12.0 provides four specific Discovery Connector nodes with this capability:

• Microsoft Azure Blob Storage Request node
• Amazon S3 Request node
• Microsoft Entra ID Input node
• Microsoft Entra ID Request node

To use an HTTP Proxy in conjunction with these message flow nodes, a Toolkit user should start by creating an HTTP Proxy policy within a policy project. The HTTP Proxy policy provides a URL (with hostname and port) for addressing the HTTP Proxy, and a Credential Name which is used to reference the credential to be used when connecting to the proxy.

Once the policy has been created and saved, and having placed one of the four supported message flow nodes in to your flow, you should then start configuring the node in the usual way by clicking the Launch Connector Discovery button. The discovery process will continue as normal (selections for a policy project and the vault in which credentials will be stored), but when it comes to connecting to the endpoint application, you will find a new optional property in the dialog named Proxy name, as shown below.

As shown in the example, this field is used to specify the name of the policy project and the HTTP Proxy Policy it contains. When clicking connect, this policy will be accessed to retrieve the relevant required connection information in order to use the proxy. The proxy will be used both at discovery time, and the configuration will also be persisted into the message flow and used at runtime when data is sent through the message flow. For this reason, remember to include the HTTPProxy policy alongside your message flow when deploying to an integration server.

For a long time, App Connect Enterprise (and its predecessor products of different names such as IBM Integration Bus) have enabled users to publish monitoring events as messages pass through particular nominated locations in a message flow, such as from particular named terminals of particular message flow nodes. Thes monitoring events have traditionally been published using IBM MQ or the MQTT protocol, but they can also be written to a local file if you prefer. Until this mod release (ACE 12.0.12.0) when writing monitoring events to a file, the product has hardcoded the file path, number of files and maximum size of each file using the following values:

• File Path: <work-dir>/config/common/monitoringEvents
• NumberOfFiles: 4
• SizeOfFile: 25MB

Starting from ACE 12.0.12.0 new options have been provided in the server.conf.yaml file which allow a user to select their own values to enable more detailed control of how this feature operates. Shown below is a relevant example stanza from the server.conf.yaml file:

Events:
  BusinessEvents: # Monitoring events
    File:
      enabled: true          # Set true or false, default false
      outputFormat: 'json'   # Set comma separated list of one or more of : json,xml. Defaults to 'json'
      #filePath: ''          # If this is set to '' then the default path is <work-dir>/config/common/monitoringEvents.
      #numberOfFiles: 8      # The maximum number of files that monitoring event file writing can rotate through.
      #sizeOfFile: 50        # The maximum size in MB of a single file that a monitoring file can use before rotating to the next file.

ACE mod release 12.0.12.0 introduces a new feature in the Web UI which can help when diagnosing unexpected flow behaviour in production environments. The purpose of the feature (when switched on, as the capability is not enabled by default) is to summarise the set of alternate paths which have been traversed by messages passing through a message flow. When working with complex flows, this provides a method of quickly drilling down in to areas of interest when troubleshooting.  For an enabled message flow, the Path analysis tab carries a tabular view showing identifiers for each path which has been recently traversed, and the Message count column shows the number of messages which have followed the given path.

There are three ways in which you can switch on the Path Tracking feature:

• Modify the server.conf.yaml to set the "enabled" property of the PathTracker Resource Manager to "true", as shown below:

  PathTracker:
    enabled: true # Enable path tracking for all message flows, default is false.

• Use the mqsichangeproperties command to set the "enabled" property of the PathTracker Resource Manager to true. For example for an integration node (named "MyIntegrationNode") owned server named "default"

 mqsichangeproperties MyIntegrationNode -e default -o PathTracker -n enabled -v true

• Use the Admin REST API to patch the "enabled" property of the PathTracker Resource Manager to "true". For example for an integration node owned server named "default":

curl -v http://localhost:4414/apiv2/servers/default/resource-managers/path-tracker -X PATCH --data "{ \"properties\": { \"enabled\": true} }" -H "Content-Type: application/json"

Once the feature is enabled, as a message is propagated through the flow it stores context information in memory which is updated each time a new message flow node terminal is encountered. For performance reasons this is stored on a per thread basis.  When the message flow completes this context information is converted into a completed path representation. At this point a per-flow cache is interrogated to see if this path has been seen before. If the path is new then the path will be added to the cache and the message number will be incremented, adding to the message list for the path in question.

In order to ensure that the cache does not consume an unbounded amount of memory, the size of the cache is limited to 20 entries per flow and for similar reasons the message list is limited to 100 message IDs per path.  Should the size of the cache reach the maximum 20 entries, and a new path be observed, then the entry which was accessed least recently is evicted from the cache in order to make room for the new entry. The message list stored the most recent 100 entries for each path, if a new message is added the oldest entry is evicted from the message list.

This has a few consequences for how the data should be interpreted. Firstly it is possible for a path to drop out of the cache and then be added back to the cache at a later time because that path hasn't been traversed for a long time. At this point its message list will have been lost and message count reset, so the message counts should be viewed as a relative measure of path usage rather than an absolute number of messages processed. 

More detail on this exciting new feature including a full worked example is provided in a dedicated blog entry which was recently published by David Crighton, Lead Architect for the App Connect Enterprise Level 3 Service team.

ACE 12.0.12.0 continues our mission to enable more functionality of the integration server to work when using Java 11. You can create a work direcory for an independent integration server and then configure it to use Java 11 using the following commands.

Create a working directory:

mqsicreateworkdir C:\temp\MyWorkDir
mqsicreateworkdir: Copying sample server.config.yaml to work directory
        1 file(s) copied.
Successful command completion.

Configure to use Java 11:

ibmint specify jre --work-directory C:\temp\MyWorkDir --version 11

If you navigate to the work directory of the server, you will find a server.java.yaml file will have been created which redirects to another file named embedded.java.yaml in the product's installation directory which specifies which java functionalities should remain disabled for the server. For ACE 12.0.12.0 the contents of this file are as shown below:

ACE 12.0.12.0 introduces a small enhancement to the App Connect Enterprise vault technology. An App Connect Enterprise vault can be used to symetrically encrypt and store credentials, which can then be used to access secured resources and endpoints from a message flow. Prior to this enhancement, a product administrator with local command console access on the machine where ACE is running, who is also in possession of the vault key, could execute the mqsivault command with a decode option if they wanted to retrieve named records from the vault and have them displayed on screen. Consider the following example.

Create an external directory vault:

mqsivault --ext-vault-dir C:\temp\MyExtDirVault1 --ext-vault-key SuperSecret123 --create

Add a credential to the external directory vault:

mqsicredentials --ext-vault-dir C:\temp\MyExtDirVault1 --ext-vault-key SuperSecret123 --create --credential-type ftp --credential-name MyFTPIdentity --username ben --password myftppassword

Retrieve the credential from the vault and display it again:

mqsivault --ext-vault-dir C:\temp\MyExtDirVault1 --ext-vault-key SuperSecret123 --decode credentials/ftp/MyFTPIdentity

... and the credentials will be displayed like this:

Namespace: credentials
Record: ftp/MyFTPIdentity
{"name":"MyFTPIdentity","type":"ftp","properties":{"authType":"basic","password":"myftppassword","username":"ben"}}
BIP8071I: Successful command completion.

Some users of ACE might not want the system administrator to have such wide ranging powers, so from this latest mod release 12.0.12.0, when creating a vault you can specify --vault-options no-export and this will prevent the retrieval, display and export of the credentials in a vault. Note that this request can only be specified when the vault is being created, and you cannot change options after the vault has been created. Compare the following example with the one above:

Create an external directory vault but this time use  --vault-options no-export:

mqsivault --ext-vault-dir C:\temp\MyExtDirVault2 --ext-vault-key SuperSecret123 --create --vault-options no-export

Add a credential to the external directory vault just like before:

mqsicredentials --ext-vault-dir C:\temp\MyExtDirVault2 --ext-vault-key SuperSecret123 --create --credential-type ftp --credential-name MyFTPIdentity --username ben --password myftppassword

Attempts to retrieve the credential from the vault and display it again will not work:

Namespace: credentials
Record: ftp/MyFTPIdentity
BIP15178E: Entries from this vault cannot be displayed due to vault options.
The vault was created with an option to prevent decode of its entries.