Introducing the Author
I’m Dave Arnold, Chief Architect for the IBM Hybrid Cloud Integration Business Unit in Australia and New Zealand. Originally from the IBM Hursley Development Lab in the UK, I have spent the majority of my 33 years with IBM in the field working with clients, solving integration and messaging problems. For the last 15 years I have led the technical sales team for middleware down here in Australia.
Looking specifically at the last 3 years, there has been a common thread in play across the conversations I have been having with clients of all sizes and from all industries. That common thread can best be summarized as….
“Our organization is on a journey to cloud and digital transformation. How can we derive greater flexibility and agility in the integration and messaging space? Our digital teams are innovating at speed and our integration team and platform can’t keep up.”
In this article I’ve tried to capture the salient points from many of those conversations and articulate IBM’s vision for Integration modernization.
If you are not a fan or reading articles, jump the end and follow the link to take in my video presentation on this subject.
The Analysts’ View – Capturing our Clients’ Sentiments
Why is integration so important to organizations on a Digital Transformation journey?
Leading analysts Gartner and Forrester have captured the sentiments of many of our clients, that integration modernization is a fundamental part of digital transformation.
“Integration has become an obstacle to success… Traditional approaches cannot cope with the volume and pace of business innovation.” – Source: Gartner – Modernizing Integration Strategies and Infrastructure Primer for 2018
“To drive new customer experiences organizations must tap into an ever-growing set of applications, processes and information sources – all which significantly expand the enterprise’s need for and investment in integration capabilities – Source Forrester Research”
Certainly, the conversations I have with clients (both with IBM Integration solutions and other vendor offerings) on a daily basis echo those sentiments.
Modern platforms, DevOps tooling and agile approaches have accelerated the rate at which organizations can bring new business function to bare. Add to that the adoption of new SaaS offerings and the ability to shift on premises systems to public cloud infrastructure and we find integration teams using traditional patterns, platforms and constructs such as ESBs struggling to keep pace.
Changing the discipline of Integration – Architecture, Technology, People and Process
So, integration as a discipline within the organization needs to change, and at IBM we see no reason why integration teams cannot derive many of the benefits being enjoyed by developers in leveraging DevOps tools on modern container based platforms to deliver agile integration.
Technology alone can help but we also need to consider new architectural approaches and changing the way our teams work or perhaps are even organized.
In a nutshell, IBM’s vision for integration modernization encompasses:
- Architecture
- Technology
- as well as People and Processes
Figure 1. Architecture, Technology, People and Process
Architecturally we have aligned our technologies for cloud native deployment allowing for the exploration of new architectures such as microservices architecture where the focus is on delivery and in the space of application integration, we can look at more fine-grained deployment of integration services when and where they are needed.
In the Technology space. IBM Integration products have been modernized to leverage the operational agility of cloud native infrastructure plugging into the core platform services to deliver a common look and feel across the portfolio.
And finally looking at People & Processes. With a delivery focused architecture and a common approach to deployment over standards based dynamic infrastructure we can consider decentralizing the discipline of integration itself. Delivering and iterating through integration services at pace without affecting other services.
So, to summarize: Agile Integration is achieved when we combine architecture, technology, people and process and align them to three modernization aspects of:
- Architecture – Delivery-focused Architecture
- Technology – re-engineered middleware on a cloud native platform
- People and process – Decentralized ownership
and apply them to our integration discipline across the Application Integration, API Management and messaging/events landscape.
Figure 2. Modernization with Agile Integration
Let’s take a closer look at each of these areas starting with the architectural aspect first.
Agile Integration – Delivery Focused Architecture
This architectural approach to delivering integration shares its roots with microservices architecture where we look to break up the monolith of the centralized ESB for an integration layer and look to make the deployment more delivery focused. For example, in the space application integration space this would be individual integration services. These services will leverage messaging as required rather than as a dependency and feature a business focused interface rather than being backend system aligned.
Cloud-native infrastructure allows us to leverage common platform features rather than product specific ones for HA, scaling and load balancing for example.
Figure 3. Architectural Approach across integration aspects
I can recommend reviewing my colleague, Kim Clark’s excellent series or articles and related collateral on Agile Integration: New paper on lightweight, agile integration architecture
As well as the official IBM Agile Integration site: https://www.ibm.com/cloud/integration/agile-integration
And watch out for a comprehensive IBM Integration Modernization with Agile Integration Redbook coming your way soon.
Agile Integration – Technology
Middleware Technology for the Modern Era
IBM Integration and messaging technologies have been on a journey of re-engineering to align to container and more recently Kubernetes based deployments. We started that journey in 2015 with production support for IIB and MQ in containers and then over three years delivered App Connect Enterprise v11 and MQ v9.1, engineered as cloud native apps and packaged for cloud native Kubernetes deployment.
API Connect, in transitioning from V5 in VMs to the 2018 version for Kubernetes followed that trend. IBM Event Streams our Kafka event hub was cloud native from the outset, whilst DataPower has enjoyed support in containers for many years. And of course, we have Event Streams, MQ, App Connect Enterprise, and API Connect managed cloud services that run on the IBM Cloud Kubernetes Service, and in the case of MQ, on AWS’ EKS as well.
This approach of standardizing our middleware technology on Kubernetes has allowed us to deliver a Hybrid Integration Platform that provides a common experience and set of tools regardless of whether you are leveraging our iPaaS services, running a traditional ESB integration construct or a modern, agile, delivery focused integration architecture on premises, or on public cloud infrastructure.
Figure 4. Hybrid Integration Platform
Our Hybrid Integration Platform is branded as The IBM Cloud Pak for Integration, ICP4i. More on ICP4i later.
A Cloud Native Platform for Integration Modernization
To deliver Integration Modernization we need a cloud native platform that can support this cultural shift in how we are going to deliver integration.
Kubernetes is that platform. Organizations are adopting container-based solutions with Kubernetes orchestration to:
- Reduce development and operating costs
- Improve application delivery
- Make more efficient use of infrastructure
We have organizations reporting savings in dev/test costs, production costs and operating on significantly fewer servers.
At IBM we see no reason why integration cannot “have a piece of this action”. This is why the IBM middleware portfolio has been consolidated to a Hybrid Integration Platform aligned to run on Kubernetes and tap into these savings and efficiencies.
Agile Integration – People and Process enabling De-centralized Integration
These first two aspects coupled with the re-engineering of the IBM portfolio as lightweight, standards- based cloud native applications provides the opportunity for de-centralized ownership where integration can become part of the delivery of a business function itself rather than a discipline found centrally, outside of the scope of the application team delivering that business function.
However, based on the conversations I am having in the field, this shift in ownership and the organization of delivery teams will be a balancing act, actually an optional transition over time rather than an all or nothing leap.
Figure 5. Decentralization of Integration
Many of the organisations I have been speaking to intend to take a foundational step to modernization where they will upgrade to the latest version of the IBM middleware, move to cloud native infrastructure and simplify their approach to non-functional requirements by leveraging platform rather than product features. They plan to shift from a high dependency ESB pattern to lower dependency delivery focused architecture for integration BUT integration as a discipline and skillset will remain as a core IT function, initially at least. These clients recognize that the investment they have made over time in using IBM integration products is not written off. There is a great deal of integration and data transformation logic that can be harvested from existing ESB deployments and redeployed in this delivery focused manner as more independent services. IBM’s longstanding commitment to forward compatibility at a developer tooling and runtime engine level ensures this is possible even when switching from a heritage zOS native, pSeries LPAR or Win/Lin VM to a cloud native Kubernetes deployment.
You’ll have heard the analogy of “cattle and pets”. This foundational step is something in between. A goldfish perhaps, lives in the house, you give it a name, feed it, keep the water clean but if it dies you quickly get another that looks the same and don’t tell the kids. Alternatively, its Daisy the favourite cow. Lives in the field with the other cows, looks like a cow, smells like a cow but the kids gave it a name therefore it enjoys, shall we say “special privileges” that are not afforded to the rest of the herd.
Introducing IBM’s Platform for Integration Modernization – IBM Cloud Pak for Integration
The modern era calls for integration to be hybrid in a number of dimensions. An integration platform should be deployable any where and exhibit the same developer, administrator and operator experience. It should also address integration requirements that themselves are hybrid in that they span the full gambit of the cloud landscape, connecting SaaS applications, traditional on-premises workloads and workloads that have been modernized and/or moved to cloud infrastructure.
A “Hybrid Integration Platform” is something you build in order to deliver integration capabilities across the broadening cloud landscape from traditional on-premises integration, modern agile integration and leveraging Integration Platform as a Service (iPaaS) capabilities.
Many of the vendors in the integration market badge their offerings as a Hybrid Integration Platform. But basically they have taken their VM based ESB and API Management monolith unchanged, spun it up on AWS EC2 and then paid an army of offshore SMEs to feed and water it for clients.
IBM have taken a different approach. As mentioned earlier our component middleware products:
- IBM App Connect Enterprise
- IBM MQ
- IBM Event Streams
- IBM DataPower (Virtual appliance)
- IBM API Connect
- IBM Aspera
have been re-engineered for cloud native deployment on Kubernetes.
We have crafted a common user experience across the portfolio that plugs into the standards-based management, logging and monitoring tools, ElasticSearch, Logstash Kibana, Prometheus and Graphana open source capabilities baked into the cloud native platform for all components.
We deliver this Hybrid Integration Platform experience as the IBM Cloud Pak for Integration which gives clients a catalog of capabilities ready to be deployed to a Kubernetes runtime across the cloud landscape.
Figure 6. IBM Cloud Pak for Integration
So, what is this cloud native platform with Kubernetes runtime of which I keep referring to? There are no prizes on offer for answering that one. It’s the Red Hat OpenShift Container Platform, RHOCP and here’s why.
Yes, RHOCP is market leading and pervasive but beyond that, neutrality across infrastructure is key in delivering the IBM Cloud Pak for Integration. RHOCP is available for on premises deployment behind the firewall or on any public cloud IaaS. There are managed services for RHOCP offered on AWS, Azure and of course IBM. The correlation between the OpenShift container platform and the Red Hat Enterprise Linux (RHEL) operating system enables consistency of operations from a reliability, security and availability point of view. Therefore, we have certified the ICP4i middleware for RHOCP creating our container images based on the Red Hat universal base image: https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
This consistency gives us a more predictable environment and makes management in general easier by enabling us to leverage a core set of common services delivered by the platform.
The IBM Cloud Pak For Integration populates the image repository and service registry being leveraged by the platform with IBM middleware product images and helm charts that have been built and tested by IBM product experts to best practice in our labs.
Deployment of IBM middleware via the IBM Cloud Pak for Integration gives us a supported and consistent approach across multiple cloud deployment models.
Figure 7. Cloud Pak Explained
Some might point the finger and say this is a licensing bundle of IBM middleware, but in truth it’s a single platform of integration and messaging technology delivered as a catalog of capabilities on RHOCP.
It is true to say, there is not one runtime BUT modern micro services architecture says that lightweight runtimes performing specific tasks, exposed as APIs, independently deployed, managed and scaled is the way forward. We are completely aligned to that philosophy. And with a common centralized User Experience developers, administrators and operators are not impacted.
Licensing is by “common currency” the ICP4i virtual processor cores have an exchange rate to the deployed core count of the component products at runtime.
The ICP4i entitlement allows clients to run any mix of all the IBM integration and messaging middleware products plus entitlement to the Red Hat OpenShift Container Platform as it’s runtime!
The Licensing model also supports a transition over time, from traditional IBM middleware deployments to a cloud native model. Where existing entitlement to currently deployed IBM Middleware can be covered by the ICP4i license with no requirement to re-platform any or all of that deployment. This affords clients the flexibility to change their integration and messaging product “mixes” as their requirements change and re-platform to cloud native as their cloud strategy evolves.
More details are available on the official IBM site: https://www.ibm.com/cloud/cloud-pak-for-integration/get-started
Where you’ll find product tours, the ability to try it out and the ICP4i documentation.
Want to take a look at an IBM Cloud Pak for Integration demo?
Figure 8. Demonstration – Retail Example
It’s an online shopping scenario where a customer on a mobile app can shop for items simply by taking a picture of them. Behind the scenes the Cloud Pak for Integration has deployed a messaging and event service via IBM MQ and Event Streams, application integration services via IBM App Connect Enterprise and exposed APIs to those services for consumption via the mobile app using API Connect.
In this example, those services are deployed via the Cloud Pak for Integration navigator into a Kubernetes runtime that was based on IBM Cloud Private.
Not a fan of reading articles?
Click on the image below to see IBM’s Vision for Integration Modernization in 20 mins – video presentation.
Hopefully, you’ve found this material interesting and relevant.