With the availability of IBM Integration Bus v10 fixpack 6 you are now able to exploit the additional connectivity provided though the new JavaScript LoopBackRequest node.
The LoopBackRequest node allows you to implement synchronous CRUD, create, retrieve, update and delete, operations with a range of data sources such as NoSQL databases. LoopBack provides numerous connectors to backend data systems such as:
In this post I’ll walk though the steps to install the LoopBack connector for MongoDB and show how to configure and then perform CRUD operations on a data collection in MongoDB from a REST API implementation using Message Maps to graphically define the details of the operations performed by the LoopBackRequest node and process the output from the it.
Setup
Installing the MongoDB LoopBack connector into IBM Integration Bus runtime
IBM Integration Bus v10 fixpack 6 now includes the “npm” Node Package Manager, command line tool to enable the installation of LoopBack connectors into your IBM Integration Bus installation to enable connectivity to a Data Source via the LoopBackRequest node.
The steps for “Installing the LoopBack connector” require that you open a IBM Integration command console and change directory to the IBM Integration workpath “node_modules” sub directory and issue the “npm” command.
For the MongoDB LoopBack connector on Windows this will be as follows:
C:>cd %MQSI_WORKPATH%\node_modules
C:\ProgramData\IBM\MQSI\node_modules>npm install loopback-connector-mongodb --save
`-- loopback-connector-mongodb@1.15.2
+-- ...
| `-- ...
+-- loopback-connector@2.4.0
`-- mongodb@2.2.7
+-- ...
Installing MongoDB and sample data
If you do not have an existing installation of MongoDB refer to the MongoDB web site for details.
In this post I’ll use the Restaurants Collection sample data set provided on github.
Having installed MongoDB and downloaded the Restaurants Collection sample data to a file “primer-dataset.json” import the data set as detailed here, for example on my local Windows installation:
c:\Program Files\MongoDB\Server\3.2\bin>mongoimport --db test --collection restaurants --drop --file C:\devel\WBI\LoopBackRequest\primer-dataset.json
2016-08-23T09:28:03.054+0100 connected to: localhost
2016-08-23T09:28:03.056+0100 dropping: test.restaurants
2016-08-23T09:28:04.019+0100 imported 25359 documents
This will drop any existing “restaurants” collection in the “test” database and load it from the file.
Configuring the MongoDB data source for use in IBM Integration LoopBackRequest nodes
As detailed in “Configuring the data source and models for your LoopBack connector” the next step is to create the “datasources.json” and setup a stanza in it for MongoDB.
For your IBM Integration installation the “datasources.json” file will be loaded from the “loopback” folder within the “connectors” folder in the Integration Bus node’s “workpath”.
For example on my Windows system:
C:>cd %MQSI_WORKPATH%\connectors\loopback
C:\ProgramData\IBM\MQSI\connectors\loopback\
| datasources.json
The properties that you can configure for each LoopBack connector you install are defined by that connector. The full set of properties supported by the MonoDB LoopBack connector are detailed here.
The minimum set of properties required for an IBM Integration LoopBackRequest node to interact with the “test” database we have populated with the “restaurants” collection is shown below:
"mongodb": {
"host": "myhostname",
"port": 27017,
"database": "test",
"name": "mongodb",
"connector": "mongodb"
}
Where “myhostname” should be replaced with the hosthame the MongoDB server will be running on and “27017” should be adjusted if you are running MongoDB on a none default port.
Note that in this use case there is no requirement to provide a LoopBack model definition for the MongoDB restaurants objects. Refer to the post “Using some of the more advanced features of the LoopBackRequest node” for an example of working with LoopBack model defintions.
Securing the MongoDB data source connection for LoopBackRequest nodes
The properties that you can configure for the MonoDB LoopBack connector in the “datasources.json” file do include “username” and “password“. However the values provided there would be in clear text, and would be applied to all LoopBackRequest nodes deployed in your IBM Integration node.
Instead you can utilize the “mqsisetdbparms” command to store a “username” and “password” with a name “loopback::securityIdentity” so that this can then be associated with the connection for a specific LoopBackRequest node by specifying a matching “securityIdentity” on it’s “Security identity” property
For more details see “Specifying security credentials for connecting to a secured data source“.
Having a separate security identity for different LoopBackRequest nodes allows you to have for example an unsecured message flow that has LoopBackRequest nodes specifying a “securityIdentity” that is configured with a MongoDB user that can only retrieve records. While another message flow which is secured with message flow security has LoopBackRequest nodes specifying a “securityIdentity” that is configured with a MongoDB user that can also create, update and delete records.
Using the LoopBackRequest node
Configuring the LoopBackRequest node in a message flow
The LoopBackRequest node can be simply added to a flow and configured statically to retrieve all records.
Figure 1. LoopBackRequest node retrieve records
After adding the LoopBackRequest node to the message flow set the following properties:
- Data source name
Enter the same name used when creating the data-source stanza in the datasources.json file
- Object
The type of object that you want to access in the LoopBack connector, for MongoDB this is the name of the collection within the Database that has been specified in the “database” property in the data-source stanza in the datasources.json file. For this example we’re using the “restaurants” collection.
- Operation
The operation defaults to “retrieve”
- Security identity
The “securityIdentity” name when accessing a secured MongoDB and having run “mqsisetdbparms” command.
Note that the LoopBackRequest node operations are synchronous and non transactional, meaning that if a message flow fails and rolls back after the LoopBackRequest node the operation on the data source will still complete.
The LoopBackRequest node also publishes LoopBack activity log events for each operation.
Controlling details of operations performed by a LoopBackRequest node
The business processing requirements of your message flow will normally need you to control details of the data operations performed by a LoopBackRequest node, for example selecting only records that match some criteria. The data operations of the LoopBackRequest node can be controlled dynamically though the LoopBackRequest LocalEnvironment settings.
To demonstrate this capability in this post I have implemented a integration solution that exposes a REST API which will have operations that set the data operation and selection criteria via query parameters by use of a Mapping node with a Graphical Data Map to take values from the query parameters if they are present and set control values into the appropriate fields in the Local Environment to control the LoopBackRequest node.
The REST API operations return a simple JSON object that needs to be populated from the entries retrieved from the MongoDB “restaurants” collection via the LoopBackRequest node. A Message Map requires a model for the data that it will transform. The data model(s) used by the REST API will be obtained by the map from the REST API’s swagger document. We will also require a model of the JSON data that the LoopBackRequest node outputs and inputs when retrieving and creating / updating entries in the “restaurants” collection. One way to provide this model is to create a JSON schema for the message maps. If you do not have a JSON schema for the MongoDB data records you could consider using one of the publicly available tools to generate a JSON schema from a sample JSON data instance, such as JSONSchema.Net. In this application I created the “restaurants.json” JSON schema to provide the model of a single restaurant object and an array of restaurants.
In the following I detail the REST API operations that I have implemented in the example to demonstrate how to achieve the CRUD operations with the MongoDB database via LoopBackRequest node. The sub flow for many of the REST API operations are composed of three message flow nodes, Mapping; LoopBackRequest; Mapping, that follow the pattern detailed in Implementing a REST API operation with intermediary processing by using message maps.
Figure 2. Using Message Maps with LoopBackRequest node
For the initial Mapping node the input is the REST API operation, and it’s output is to populate the required settings for the LoopBack request node, which is often just an empty message body, modeled using the BLOB message, and settings in the Local Environment. To understand how to add the LocalEnvironment to a Message Map see this topic. The Mapping node after the LoopBackRequest node is provided to transform the MongoDB data records, using the “restaurants.json” JSON schema into the response format of the REST API operation.
I have provided the LoopBackRequest MongoDB_RESTAPI project interchange for this solution.
Creating a Restaurant record
The “postRestaurant” REST API operation takes a “NewRestarant” JSON object which is used by the message map “postRestaurant_PopulateRestaurantRecord” to build a MongoDB “restaurant” record, defined in the “restaurants.json” JSON schema.
The LoopBackRequest node does not require any LocalEnvironment data in this case and just uses the values set on it’s properties.
The output of the LoopBackRequest node for a successful create operation is the JSON data for the created record. In the demonstration REST API this object is simply returned by the operation sub flow.
You can use tools such as Swagger Editor to import the REST API “swagger.json” and test this operation.
Retrieve Restaurant entry by unique id
The “getRestaurant” operation takes a single required parameter which is the “id” value of an entry in the MongoDB “restaurants” collection. The “getRestaurant_SetIdForLoopBackRequest” message map simply Moves the input REST API parameter “id” into "LocalEnvironment.Destination.Loopback.Request.id".
The operation could be invoked via
http://localhost:7800/loopbackrequest/v1/MongoDB/Restaurant?id=57bc091316af78716fed54ee
assuming the REST API was deployed to a local Integration node. The value of the “id” must be set to match an actual “ObjectId” for an entry in your MongoDB instance. If the “id” does not match any record the LoopBackRequest node will complete successfully, returning an empty output message body.
When the “id” value provided does match a record in the MongoDB collection the operation response provides the following JSON data object, created by the Message Map “getRestaurant_MapRestarantResult” which has it’s input defined from the JSON restaurant object in “restaurants.json” and output defined by the REST API swagger document. The “maxGradeAScore” is computed using fn:max($Item[grade = 'A']/score):
{"name":"21 Club","cuisine":"American ","borough":"Manhattan","maxGradeAScore":12}
Retrieve matching Restaurants entries and controlling the number of records returned
The “getRestaurants” operation takes a number of optional parameters, “match” which allows selection of which record(s) to return, “limit” which can control the maximum number of records in the output. To achieve pagination the operation also offers a “skip” parameter.
In the operation sub flow the “getRestaurants_SetUpLoopBackRequestFromParams” message map provides the following logic:
Retrieving the “top 3” entries by controlling match and order
The “getTopRestaurants” operation takes a “cuisine” parameter and an optional “borough” parameter and computes a selection to enable it to return an ordered list of the names of the top three scoring Restaurants providing the specified “cuisine” and optionally being located in the provided “borough”.
This is achieved by the “getTopRestaurants_SetupOrderedRetrieve” map in the operation subflow using the simple input REST API parameter values to build LocalEnvironment settings for the LoopBackRequest node. For example when selecting the top scoring “American” cuisine restaurants in “Queens” these REST API parameter values are mapped to populate the following local environment tree data to control the LoopBackRequest node:
LocalEnvironment
Destination
Loopback
Request
filter
where {"and": [{"cuisine":{"eq":"American "}},{"borough":{"eq":"Queens"}}]}
limit 3
order
grades.grade DESC
order
grades.score DESC
field
name true
- filter.where
The map uses an If to test whether “borough” parameter is provided.
When just the “cuisine” parameter is present the “where” is created as
{"cuisine":{"eq":"$cuisine"}}
using a fn:concat XPath function.
When both “cuisine” and “borough” parameter values are passed the else branch creates a “where” with an “and” clause as
{"and": [{"cuisine":{"eq":"$cuisine"}},{"borough":{"eq":"$borough"}}]}
to select only records that match both “cuisine” and “borough”
- filter.order
The REST operation is required to order the restaurants matching the where clause by both the grade level and score to provide the top rated restaurants. This requires that two “order” array entries are created. In the graphical data map the Append transform is required to create instances of an output array when there is no repeating element on the input side. The Append requires an input connection for each instance of the output array needed. In this case we actually just want to assign a fixed literal “DESC” for descending order, but have to make “unused” connections from the “cuisine” parameter to provide these required input connections. The child of the “order” array element needs to be created with the name of the field in the record to order by. To achieve this use the mapping add user-defined function to create the elements with names “grades.grade” and “grades.score” to signal that we want the records ordered by the “grade” and “score” sub elements of “grade”.
Figure 4. Mapping to setup a top three LoopBackRequest
- filter.field
Since this REST API operation will just be returning the “name” of the restaurants in our example, the map is also creating a filter of the fields that will be returned in the results records to be just the “name”. This is achieved by creating a user-defined element called “name” and assigning it’s value to “true”. This means that the only field in the returned data will be the name, avoiding the overheads of passing back a complete record.
Updating Restaurant records
The “putRestaurants” operation takes a JSON object which provides “name” and “borough” to identify restaurant record(s) to update with a “grade” and “score”.
The “putRestaurants_SetupUpdate” message map uses the “name” and “borough” to create a where clause in LocalEnvironment.Destination.Loopback.Request.filter.where as {"and": [{"name":{"eq":"$name"}},{"borough":{"eq":"$borough"}}]} using a fn:concat(), this will cause the update to be applied to the restaurant record(s) to be updated. The output body of the map is set as the “restaurant” JSON object type from the “restaurants.json” JSON schema file. The map moves the “grade” and “score” into this to provide the partially populated update.
The LoopBackRequest node returns the result from the installed connector, which in this case is a simple JSON object containing a “count” of the number of records matched and updated. Note that if no records match the operation is still considered successful and the return is a count of zero.
Delete a Restaurant record
The “deleteRestaurant” REST API operation takes a single “id” parameter which is used by the message map “deleteRestaurant_SetId” to set the LoopBackRequest “id” LocalEnvironment value.
In this case if the passed “id” value does not match any record in the MongoDB restaurants collection then the LoopBackRequest node will raise an error like:
BIP3873E: The 'Delete' operation was sent to the configured LoopBack connector for object 'restaurants' with LoopBack ID 'noSuchId', but no record could be found with that ID.
and the REST API operation will return a failure based on this.
Deleting Restaurant records
The “deleteRestaurants” REST API operation takes a single “match” parameter which is used by the message map “deleteRestaurants_SetupDelete” to set the LoopBackRequest “externalIdName” and “externalId” LocalEnvironment values in a simlar manner to that detailed in the “getRestaurants” operation above.
The LoopBackRequest node returns the result from the installed connector, which in this case is a simple JSON object containing a “count” of the number of records matched and deleted. Note that if no records match the operation is still considered successful and the return is a count of zero.
Summary
In this post I have reviewed the capabilities of the LoopBackRequest node and provided an example of how to use it to perform CRUD operations on a data collection in a MongoDB database using message maps in a REST API implementation.
I briefly presented the use of the install of the LoopBack MongoDB connector and provided details of the configuration that is required to prepare the Integration Bus LoopBackRequest node to operate with a MongoDB database.
I then described the usage of the LoopBackRequest node and detailed using message maps to populate the LocalEnvironment settings that can dynamically control the operation performed by it to achieve the desired actions for an example REST API.
Hopefully this post will enable you to understand how to exploit the capabilities of the LoopBackRequest node in your own IBM Integration solutions. Please post any comments or questions as a comment to this post or at dW Answers Integration Bus.