Message Image  

Graphical data mapping with a Global cache

 View Only
Tue July 14, 2020 12:16 PM

With the availability of IBM Integration Bus v10 fixpack 2 it is now possible to use simple Graphical data mapping to access Global Cache key-value data.

The global cache is a repository for data that can be efficiently reused. For example, you can use a global cache to store frequently accessed lookup tables, or state correlation information. The cache data can be for used beyond the scope of a specific message flow node, instance of a message flow, integration server, or integration node. The cache facilitates sharing of data across processes (both in the same integration node, and across integration nodes) and eliminates the need for an alternative solution, such as a database. You can use one message flow node to store data in the global cache, then a second node (in the same message flow or a separate flow), can retrieve that data from the global cache.

Prior to IBM Integration Bus v10 fixpack 2 the message flow note that interacted with the global cache had to do so programmatically though use of the Java user-defined extensions API class MbGlobalMap. While this could be invoked from a Mapping node using a Custom Java transform, the development and support would required Java skills.

From IBM Integration Bus v10 fixpack 2 the Graphical Data Map now includes three new transform types:

These transforms allow you to simply visually interact with the global cache to read, write and remove simple type key-value data entries.

The Tutorials Gallery included with the IBM Integration Toolkit, accessible from the Welcome page and from the Help menu, now includes the additional tutorial “(V10.0.0.2) Using a Mapping node to graphically access a Lookup table that is stored in the Global Cache” that has been published in the OT4I GitHub repository “mapping-cache-tutorial”. Note that you must update your installation to v10 fixpack 2 or higher before loading and running this tutorial.

The tutorial demonstrates both retrieving a value from the global cache by using a Cache Get transform and loading key-value pairs using a Cache Put transform. The tutorial shows how the Cache Get can detect if the global cache returned a value for the given key, to enable the flow logic to branch and read a comma separated values, CSV, file from which to load the global cache via the Cache Put transform.

The Cache Get, Put and Remove transforms use nested mappings to allow the full range of transform types to manipulate the input parameters and any return data from the cache operation. All of the transforms have a nested mapping for input, while the return is only automatically provided when adding the Cache Get to a map.

Figure 1. Cache Get Transform

The Cache Get nested mapping allows you to provide values to select the key, and optionally provide a map and cache name to address the value to be retrieved from the global cache.

Figure 2. Cache Get Input

The “Cache Return” nested mapping allows you to apply any transformation to the value retrieved from the global cache and set it in the output of the mapping node. Note that you can test for the existence of the value retrieved to enable conditional processing depending on whether a match was found in the global cache.

Figure 3. Cache Get Return mapping

The Cache transforms also allow you to optionally add a “Cache Failure” nested mapping which can be used to catch and graphically process any exception encountered when performing the cache operation.

Figure 4. Adding optional Cache Failure Mapping

Note that as with the MbGlobalMap Java API the new Graphical Data Map Cache transform interactions with the cache happen outside the message flow transaction, and are committed immediately. If an exception is thrown downstream of the node that interacts with the cache, the cache interactions are not rolled back.

In summary the new capability provided in the Graphical Data Mapping editor allows you to simply exploit the full capability of globally cached key-value data as part of your visual transformation development.

2 comments on"Graphical data mapping with a Global cache"

  1. GANESH KUMAR August 27, 2016

    In one of the scenario I’m using Embedded Global Cache. I can able to perform the operations on GC. My GC is holding some of the database records. My question here is what if suppose database has been updated with some other external system. How can I populate new records in GC? I’m using Mongo as a backend through rest api.

    Any suggestions would be greatly appreciated !!

    Reply (Edit)
    • martin_b August 31, 2016


      The Cache Put transform supports “insert or update” mode which you could use to update / add records in the Global Cache when the MongoDB collection is inserted to. Similarly when there is a delete in the MongoDB collection you can use the Cache Delete transform to remove the corresponding entry.

      The challenge here will be getting the “trigger” from MongoDB that the collection is modified.

      For conventional SQL databases IIB provides the Database Input node to enable you to build a flow to respond to database updates.

      MongoDB does not directly provide “database triggers” however you could look into using “capped collections” and/or “tailable cursors” as a way of obtaining a trigger to initiate some action to drive a message flow that re-synchronises the Global Cache table with the MongoDB collection.

      Another approach if your need to pickup changes from the MongoDB collection can be non-coordinated would be to set an appropriate “time to live” value when you load the cache table. Then when the “time to live” period expires the Map can be implemented to detect that the entry is not present in the Global Cache and branch into a nested mapping to reload the Cache. Obviously with this approach there is a balance between how often the Cache entries are expired and reloaded and how long “stale” data might be retrieved from the Cache.

      While mentioning integration with MongoDB please be aware that with the With the availability of IBM Integration Bus v10 fixpack 6 you are now able to exploit the additional connectivity provided though the new JavaScript LoopBackRequest node, which can be used with the LoopBack connector for MongoDB.

      Reply (Edit)