Explore the new features in App Connect Enterprise version 11.0.0.7

App Connect Enterprise has provided KafkaConsumer and KafkaProducer message flow nodes for several years now. Fix pack 7 builds upon these capabilities with several enhancements to this area of the product. As a quick reminder, Apache Kafka is an open source distributed publish-subscribe messaging infrastructure. The Kafka cluster of brokers maintains a partitioned commit log, where each partition consists of an ordered sequence of messages whose state cannot be altered after they have been created. These immuteable messages which are stored in each partition are assigned a sequential id number, referred to as the offset. The offset uniquely identifies each message within the partition. The Kafka cluster retains all published messages (whether or not they have been consumed) for a configurable time period. After this time period, the messages are discarded to free up space.

ACEv11.0.0.7 provides a new Kafka Read message flow node, which is a mid-flow node whose purpose is to read a single message from a Kafka topic. The message to be read is identified by topic name, partition number and message offset which are supplied as node properties or as Local Environment overrides. Typical uses of the Kafka Read node might be:

• In error processing scenarios to re-process an individual message which had encountered a failure while being processed in a Kafka Consumer message flow.
• In conjunction with a Timer node where a defined range of messages on a partition on a topic are to be processed.

For users who are familiar with MQ and the concept of transactions which span the behaviour of multiple separate nodes in a message flow, it is worth highlighting that even though they may look similar on the surface, the concept of the Kafka message flow node implementation is quite different. ACE makes separate connections to the Kafka broker on behalf of each separate Kafka message flow node instance. So, when considering message flow design, note that there is no “pseudo transaction logic” happening as an MQ user might imagine it. The Kafka Consumer node commits the offset immediately (and note the ConsumerGroup id will ensure that the same consumer won’t receive the same message from the same partition twice). Given this description of behaviour it should be apparent that having a mid-flow node which is capable of being dynamically directed to read (or re-read) a message from a topic at a particular offset will be helpful in achieving some architectural design patterns. The Kafka Read node’s properties are displayed in the screenshot below:

The basic properties shown above define the connection parameters which are required to connect to the Kafka broker and topic.

Another of the node’s properties defined on the Security tab shown above, is the name of a security identity, which lets you abstract away from the flow artifact, the userid and password which should be used for the connection. Previous fix packs of both ACE and IIB have used a single security identity for all the Kafka message flow nodes in all the flows deployed to any given integration server, but from this fix pack you can now configure several identities for the use of separate message flow node instances. The mqsisetdbparms command (or new mqsicredentials command) can be used to define the userid / password combination. For example:

mqsisetdbparms -w <WorkingDirectory> -n kafka::event_streams_id -u myEventStreamsID -p "3QA9xbOXUwimNbH0r0t0g5H94PZWZTh_M-q-eVxxxxxx"

Fix pack 7 also introduces the concept of a Kafka policy type which can be used to override the settings of any given instance of a Kafka message flow node:

For more information about the ways in which the Kafka nodes use settings in the Local Environment tree, please refer to the Knowledge Center.

ACEv11.0.0.7 embeds the Apache Kafka client version 2.3.0

Although absolutely not mandatory, many users are choosing to coincide their migration to App Connect Enterprise v11 with a move to a container based architecture for their integration needs. To get the maximum value from moving to a container environment, it is beneficial to be able to independently scale ACE containers from MQ queue manager containers. Against this backdrop, ACE users thinking about a move to containers will be keen to consider any product feature involving interactions with MQ. This includes MQ message flow nodes interacting with queues, publish / subscribe interactions with topics, the distribution of monitoring messages, Record and Replay, and Event Driven Architecture (EDA for short) message flow nodes. The last item in this list is the main focus of this fix pack 7 enhancement. The EDA nodes is a collective term for the message flow nodes which rely on using messages stored on the queues of an MQ queue manager to maintain their current state:

• AggregateControl, AggregateRequest and AggregateReply nodes
• TimeoutControl and TimeoutNotification nodes
• Sequence and Resequence nodes
• Collector node

Starting from fix pack 7, it is now possible to use these message flow nodes without having a local queue manager, and instead direct them towards a remote queue manager using an MQ client connection. The server.conf.yaml configuration file now includes a remoteDefaultQueueManager property which can be assigned the name of an MQEndpoint policy in the form {PolicyProjectName}:PolicyName. This policy can be used to specify the connection details for a remote queue manager instead of the usual defaultQueueManager.

The property replacementQueuePrefix can be used to define a prefix instead of the normal queue prefix “SYSTEM.BROKER”. Using this property enables you to share a single queue manager for the purpose of supporting EDA message flow node deployments on multiple integration servers. For example, if you assigned the value MYPREFIX, then:

• SYSTEM.BROKER.EDA.EVENTS would become MYPREFIX.EDA.EVENTS
• SYSTEM.BROKER.EDA.COLLECTIONS would become MYPREFIX.EDA.COLLECTIONS

As mentioned in the introduction above, the purpose of this feature is for the benefit of standalone integration servers running within containers. For this reason, by default the integration server will be stopped should the network link to the queue manager become unavailable. In a container architecture, the server’s container would then be restarted by the container framework (eg Kubernetes).

Fix pack 7 provides enhancements to the Toolkit which make it even easier to get up and running with a development environment straight after installation. From the Integration Explorer view, there is a new Integration Servers menu option for creating a local integration server. This feature quickly creates a server with its working directory defined as a project in your Eclipse workspace. This means you don’t have to leave the Toolkit experience to create a server – you can simply start developing, deploying and testing your flows without having to leave the Toolkit experience.

After choosing to create a local integration server, the dialog offers you options to choose a name and some basic port details. You can define the port numbers which the server will use for receiving administrative API requests over HTTP, for receiving runtime HTTP traffic and for connecting the visual debugger. If you prefer you can go with the default settings and let the Toolkit find available ports:

Tip: Experienced users may already be aware of this, but the integration server will not attempt to open its runtime HTTP traffic port until it is needed, so if you have not yet deployed a message flow with an HTTPInput node, then the port will not yet be taken. For this reason if you use this new feature to start up more than one local integration server (without deploying any flows) and you use the option to automatically find an available port, you may end up accidentally trying to use the same HTTP input port for multiple servers. If you make this mistake, you can always put things right by editing the server.conf.yaml file settings and restarting your integration server.

Remember, for production circumstances you should use an integration node, unless you are operating a container based architecture in which case one independent integration server per container is the best option.

App Connect Enterprise v11 fix pack 6 provided new capabilities for the encryption of credentials for the use of independent integration servers when communicating with third party systems. This blog post described the two new commands which were offered in this feature area:

• mqsivault: Before you can store encrypted credentials, you must configure an App Connect Enterprise vault. Each vault has its own vault key. You can use the mqsivault command to create or destroy a vault, to change or verify a vault key, or to retrieve credentials from the vault. The vault stores the credentials (in encrypted form), and the integration server uses them to access secured resources.
• mqsicredentials: You can use the new mqsicredentials command to configure an integration server to use encrypted credentials for connecting to a secured resource. You can use the mqsicredentials command to create, update, report, and delete credentials. You can use the command to create and report credentials when the integration server is running or stopped, but you must stop the integration server before you can update or delete credentials.

In fix pack 7, we have extended these capabilities so that if you create an integration node by running the mqsicreatebroker command, you can create a vault for that integration node by specifying either the --vault-key or --vaultrc-location parameter on the command. You can now use the mqsicredentials command to create, update, report, and delete credentials for an independent integration server or for an integration node and the integration servers that it manages. You can use the command to create and report credentials when the integration server is running or stopped, but you must stop the integration server before you can update or delete credentials. We also provide configuration for credentials using the administrative REST API, and you can view credentials by using the web user interface. Alternatively, you can still use the mqsisetdbparms command to associate credentials with resources that are accessed by an integration server or an integration node.

The WebSphere Service Registry and Repository (WSRR) product is a central repository of entities. A wide range of entities can be stored and retrieved, including user-defined concepts and definitions related specifically to web services, such as WSDL services, service interfaces, and associated policies. However, with the popular adoption of container-based architectures and microservice style design patterns, the use of API management products are increasingly being preferred to the use of a large centralised repository dynamically queryable for web service definitions. In line with this trend, we do not expect to further enhance this aspect of the product, nor do we expect widespread use of this technology, but the provision of the WSRR message flow nodes should ease the ability for IIBv10 users to migrate to ACEv11 either in advance of moving to a container architecture, or as a stepping stone to a more API-centric architectural approach.

Fix pack 7 provides App Connect Enterprise v11 with two new message flow nodes for interacting with WSRR – the RegistryLookup node and the EndpointLookup node.

Users of IBM Integration Bus may well be familiar with these WSRR nodes, which are functionally equivalent to their IIBv10 predecessors. You can configure a message flow to dynamically retrieve resources from WSRR at run time, and use / expose those resources in the message flow. You can therefore defer the decision about which resources you want to use until run time, rather than deciding at deployment time.

The adoption of containers and Agile Integration Architecture, is also causing users to pursue ACE container images which are as small as possible. The WSRR node dependency jar files amount to approximately 100MB in terms of installation footprint. For maximum flexibility we have made the WSRR nodes an optional part of the installation. On Windows there is a new page in the installation dialog, accessed by selecting the Options button as shown in the screenshot below:

The IIBv10 implementation worked in conjunction with a DefaultWSRR configurable service. This configurable service was not dynamic (it required an integration node restart in order to take effect) so providing the same configuration properties to ACEv11 through the server.conf.yaml file is a good conceptual fit (as opposed to providing a new policy type which we also considered). The example server.conf.yaml file (created by the mqsicreateworkdir command, or sample found at the location <ACE install directory>/server/sample/configuration/server.conf.yaml) contains the properties as shown below:

IBM Cloud Transformation Advisor helps you analyze and modernize middleware based applications. Until now it has focused on categorizing Java EE apps and MQ queue managers and providing guidance for modernization in these product domains. Transformation Advisor itself comes in a couple of different form factors:

• TA Local (currently in Beta)
• TA Enterprise, part of the IBM Cloud Pak for Applications

ACE Fix pack 7 provides a built-in Transformation Advisor command, which provides support for the analysis of existing IBM Integration Bus v10 integration nodes. This capability is provided out of the box with ACEv11.0.0.7. Nothing further needs to be installed. We’re providing this feature directly within ACE to make it as simple as possible to get started but in the future it is likely we will build upon this initial capability and provide the option to export the generated information into the above form factors of the IBM Cloud Transformation Advisor tool as well. This initial delivery collects information about what was deployed to your IIBv10 integration node and analyses the data for potential issues when moving your architecture to adopt containers. The Transformation Advisor tool has common concepts bridging each of the product domains. For example, each assessment measures the potential complexity of the required transformations as either simple, medium or complex. Each product defines guidance for what these categories mean:

The Transformation Advisor tool also provides a severity classification for each IIB issue it uncovers:

• GREEN (INFO) = No immediate action is required, but you may wish to be aware.
• YELLOW (WARNING) = Immediate action is probably required or advised before you proceed.
• RED (ERROR) = You cannot proceed without taking remedial action.

To execute the TADataCollector command, use the TADataCollector environment variable to define the location where logs from the command will be written, where the IIBv10 backup file will be found, and the output report will be generated. First of all, from an IIBv10 installation, take a zip file backup of your integration node using the mqsibackupbroker command. Once you have the zip file, this has all the information inside required for analysis. Next, launch an ACEv11 command console to run the TADataCollector. The screenshot below shows an example:

The command will create a report as a static HTML page. An example of the layout is shown in the screenshot below:

You can follow the links in the summary table, or just scroll down the page, to view more detailed information about the issues that have been found in each integration server:

The initial release of this tool looks for message flow node instances which may require consideration when moving to containers. In future we expect to extend the scope of the assessment to look out for more complex situations and other product features which may require an architectural rethink when moving to containers. Keep an eye out on the integration blog for more detailed studies coming soon regarding the move to containers.

mqsireloadsecurity

ACEv11.0.0.7 reintroduces the mqsireloadsecurity command, which lets you force the immediate expiry of all or some of the entries in the security cache. As a quick reminder, a security profile defines the security operations that are to be performed in a message flow at a SecurityPEP node and / or security enabled input and output nodes. In IBM Integration Bus v10, security profiles were created using the mqsicreateconfigurableservice command, but in App Connect Enterprise v11 this task uses a Security Profiles Policy:

Security profile policies can be deployed to an integration server as part of a BAR file, or copied into the overrides directory in your server’s working directory. You can configure your message flows to use a security profile through configuration settings in the BAR file as shown below:

Currently, security profile policies cannot be changed after they have been deployed. Once deployed, security profile policies are used to control security operations by the security manager at runtime. Entries are created in the security cache when a message flow with a security profile performs authentication, mapping, or authorization. The entries are valid for the length of time that is specified by the cacheTimeout property of the securitycache component after which the entries are marked as expired. You can check your cacheTimeout setting using the command mqsireportproperties . When an entry is marked as expired, it must be reauthenticated, mapped, or reauthorized with the security provider before it can be reused, and its expiry time is reset. From this fix pack release you can also use the mqsireloadsecurity command to force the immediate expiry of some or all of the entries in the security cache. Details of the command syntax can be found in the Knowledge Center. For example:

mqsireloadsecurity <IntegrationNodeName> -e <IntegrationServerName> --all-applications

mqsireportproperties

ACEv11.0.0.7 enhances the mqsireportproperties command with the ability to list more details for HTTPConnector object. For example, consider the following command options:

mqsireportproperties <IntegrationNodeName> -o HTTPConnector -e default -r

The output will look as shown below. Note in particular the section at the very bottom which shows the URL fragments which have been registered by the deployed message flow(s).

HTTPConnector
  name='HTTPConnector'
  type='Policy'
  AutoRespondToHTTPHEADRequests='false'
  CORSAllowCredentials='false'
  CORSAllowHeaders='Accept,Accept-Language,Content-Language,Content-Type'
  CORSAllowMethods='GET,HEAD,POST,PUT,PATCH,DELETE,OPTIONS'
  CORSAllowOrigins='*'
  CORSEnabled='false'
  CORSExposeHeaders='Content-Type'
  CORSMaxAge='-1'
  ConnBacklog='100'
  EnableLookups='true'
  IntegratedWindowsAuthentication=''
  ListenerAddress='0.0.0.0'
  ListenerPort='7800'
  ListenerThreads='-1'
  MaxConnections='-1'
  MaxKeepAliveRequests='-1'
  MaxPostSize='-1'
  QueueCapacity='1000'
  ServerName=''
  TimeoutSweepInterval='20'
  active
    ListenerPort='7800'
    detailed
      connectors
        Embedded
          port='7800'
          secure='false'
          urlRegistrations
            urlRegistration
              outstandingRequests='0'
              url='/flow1'
              usedByHTTPNNodes='true'
              usedBySOAPNNodes='false'
              nodes
                node
                  flowLabel='flow1'
                  nodeLabel='HTTP Input'

Previous versions of IBM Integration Bus have included an HTTP proxy servlet, which is a Java servlet that you can use in a servlet container such as WebSphere Application Server Liberty Profile or Apache Tomcat, or in a web server such as IBM WebSphere Application Server. The HTTP proxy servlet receives HTTP requests from web services client applications and replaces the support that is provided by the integration node and embedded (integration server) HTTP listeners. The bridging communication between the proxy servlet and the message flow logic is through the use of connections to MQ queues. Two common architectural patterns using the proxy servlet are shown in the diagrams below:

By using the HTTP proxy servlet in a servlet container, you can support high availability, load distribution, and access to the integration node from multiple IP addresses and ports. From fix pack 7, you can find the proxy servlet at the location
<ACE install directory>/server/tools/proxyservlet.war. It is likely that this feature will mainly be of interest to existing users wishing to migrate this feature forward to v11. Further detailed instructions will be provided in the ACEv11 Knowledge Center soon, but for now further background information about the concepts of the proxy servlet can be found in the IIBv10 Knowledge Center.