Cloud Pak for Integration

In part 1 of this series, “Good Integration Patterns Never Die – You Just Add More”, I described several challenges and identified a set of integration capabilities that solved these challenges.  Toward the end of the article, this question was posed:

…we identified the following types of integration capabilities: screen-scraping, file, EDI, messaging, application integration, Business APIs (with a security gateway), and events. What can we notice about this full list of integration capabilities?

The stated answer was that they are all still in use today.  But the complete answer should also say that almost always these capabilities are used together to provide complete integration solutions.  When looking only at individual integration styles, the expression “can’t see the forest for the trees” is applicable.  Are you looking at the trees and not seeing the forest?

This article zooms out to see the bigger integration picture.  Starting with each integration capability as a focal area we zoom out to see how that integration capability is combined with other integration capabilities to solve the full challenge.  At the close of this article I attempt to predict the next set of challenges facing integration and how these might be solved.

Solutions Use Multiple Integration Styles

Let’s revisit the previously identified integration styles and see how they are often in combination with one another.

• Screen-scraping – If a technical interface to an application is available, it will always be more robust and scalable than screen scraping. However, sometimes on older applications there is simply no choice – there is no way into the system other than the user interface, or there is no time to create one. Today robotic process automation (RPA) is used in business processes to invoke these older applications.  But, if this were the only application required for the solution, we would simply use its existing interface.  Because screen scraping is being used this implies that the intent is to obtain information from this application to be used in a larger context.  This information may need to be transformed or combined with data from other applications to provide the full solution.  The most common scenario is that this data is used by an overarching business process or application integration. This means the RPA interface is just part of a broader process or integration with many steps, involving use perhaps of application integration, business APIs, and messaging. In an ideal world we would wrap the RPA interface to make it appear like an API, event, or other type of connector to the calling integration flow such that it could be replaced later with a more robust interface without affecting the process.

• File and EDI – Many businesses continue to exchange files to do business – either directly or through an EDI approach to format and transfer the file. But consider how the file is created before being sent and how is it read when it is received.  Application integration may be used to build a file taking multiple records from various sources, format the records, build the file, and then invoke the necessary function to send the file.  On the receiving side application integration can read the rows in the file and format these appropriately to feed into the order system, inventory system, CRM or other applications as required.

• Messaging – messaging provides transaction integrity assuring the successful transfer of a request from one application to another application. The message interaction however is typically just one of application interactions needed to complete an overall scenario. Frequently each application defines formats for data elements differently.  Application integration is used to transform the data into the format required for each application and to invoke messaging to communicate these tailored messages to each of the necessary applications.  For example, an API call from the web or a mobile app to purchase a product may initiate an order. The API is verified by a security gateway, then the API invokes application integration which may obtain information from the CRM system, use messaging to place the order in the order system, invoke messaging again to update the inventory system, and yet again to invoke the billing and logistics systems while transforming the data appropriately for each invocation and response.  Event notifications are sent as the item is shipped. Messaging is fundamentally involved at the transport layer, but there are many other integration patterns at play.

• Business APIs – The letter “I” in API stands for interface. Business logic is invoked behind this interface, and while on some occasions this may be a single call to a single application (e.g. retrieve a specific data element), in other scenarios (such as the order scenario) what is behind the API is several additional integration capabilities teasing data into or out of multiple applications.  Scenarios may include the application integration, messaging, or another capability.  Most applications and databases now automatically generate system focused APIs, but these need manipulation by an integration component to become a useful part of a consumer-oriented business API.

• Events –Events recognize that something happened. Sample events vary from items such as an airplane landing to an updated data record in a back-end system.  Applications are being written using events as the trigger to take an appropriate action.  In the context of this integration discussion events may themselves be sourced from integration activities that occur in the enterprise back-end systems - for example in our order scenario events are triggered through the application integration and messaging updates.  An inventory replenishment application sees these events are decrementing available inventory and requests additional shipments when reorder levels are reached.  How is this reorder done? Probably via an API, file, or EDI transaction.

• Application integration – Application integration provides connectivity between applications transforming the data formats, handling the protocols, and routing the requests appropriately across multiple physical locations as required. Scenarios using messaging, RPA, and files in conjunction with application integration were previously described.  Application integration is also frequently invoked via a Business API and it also may act as a source for events. 

• Security gateway – The security gateway is of course part of a larger solution. It is used for securing traffic either entering/leaving the enterprise or between applications in the enterprise – regardless of physical location.  Frequent scenarios include use with API traffic, Application integration, and events.

Each integration capability plays an important role and is optimized for a particular style of integration.  Having many capabilities available each playing different roles provides a complete toolbox allowing the right tool and the right combination of tools to be used when needed.

Why Do You Care?

The scenarios described above probably seem familiar.  There are circumstances where a single integration capability solves an issue, but there are many more scenarios where multiple styles of integration are used together.  As additional integration needs surface in your environment you may find that adding new styles of integration to some of your existing constructs provides the optimal solution.

When multiple integration styles are used together being able to share assets across the toolset speeds development.  The output of one integration style is the input to another style.  For example, having application integration or messaging constructs available to an API management solution allow a consumer-oriented API to be created by easily mapping the API inputs to these called application integration and messaging assets.  But how easy is it to use these various tools together, and share assets between them?

Piecing together a toolbox of integration capabilities is challenging.  Do the various tools work together?  How can assets be shared between tools?  Where is the master copy?  How many different user interfaces must be learned?  How much capacity for each individual tool must be purchased now, and how much is needed in the future as additional projects arise?

IBM has recognized these challenges are inhibitors to your integration success and has introduced the Cloud Pak for Integration to address these concerns. 

The Cloud Pak for Integration has combines IBMs market leading integration capabilities for API Management, Application Integration, Messaging, Events, Files, and Security gateway.  But this is not a bundle of individual products.  The Cloud Pak for Integration is the product.  It provides a common user interface that crosses all the integration capabilities and allows for a consistent look and feel - one interface to learn. All the products can be run and managed consistently using a common underlying infrastructure – one set of operations skills. A common asset repository provides sharing of assets between capabilities and is the place to find the integration assets.  Furthermore, rather than needing to estimate how much of each individual capability must be purchased, with the Cloud Pak product you simply purchase “integration”.  The purchase is a total capacity for integration and allows use of any/all the capabilities in the Pak within this limit.  You may repurpose your usage between capabilities as needed.  Early needs in your program may require significant development of new application integrations and perhaps messaging infrastructure.  Later, once the core integrations are implemented the focus may shift to exposure of some APIs to a broader, perhaps external audience. You can at this point simply use the API management function that is already purchased assuming there is still some unused capacity either from the original purchase, or perhaps from the reduced utilization of the other capabilities.  No need to make another separate tool purchase.

What’s next?

Disclaimer: what follows is a prediction based on the author’s experiences – use at your own risk.

Businesses are embarking on a digital transformation bringing new business offerings to market faster, forming ecosystems across industry partners, and expanding their enterprise reach.  IT organizations are moving to a hybrid cloud model, implementing agile development methodologies, and using a microservice architecture approach to deliver solutions faster. All of this is driving an increased need for integration as assets are needed to be combined rapidly from disparate locations – even bridging business boundaries.  Bringing the integration capabilities together in a single cohesive product – Cloud Pak for Integration - is a tremendous productivity improvement and a base that can grow to support a wide range of integration needs. 

The number of projects, teams, and integrations is accelerating and relying on a centralized expert team to perform all the integration tasks is not scalable and is a bottleneck to success.  Additional developers must be able to perform integrations while maintaining the quality of the deliverables.  Using automation to remove potential error prone tasks and make the development of integration assets simpler we can reduce the required level of expertise and enable a larger pool of resources to develop integrations.   Adding artificial intelligence (AI) to guide the non-expert developer to do things the right way based on the expertise of the central team also helps scale resources without relying on the central team to do all the work.  Prediction: Automation and AI will be the next big integration breakthrough technologies to help move integration passed this current challenge.

If you have questions, please let me know.  Connect with me through comments here or via twitter @Arglick to continue the discussion.  

Graphics courtesy of Pixabay.com – Mike Goad, Mohamed Hassan, kalhh, Tumisu, and nvodika,