How does IBM need to change their integration thought process to ensure that current traditional integration products, like MQ and IIB, can smoothly move to the cloud era? Is this even possible?

** 
This discussion is part of the new Featured Discussion program, which offers open-ended discussion group conversations that enable participants to engage around issues that are slightly broader than traditional group message topics. 

If you are interested in submitting a topic for a future Featured Discussion, please email support@imwuc.org.   

Hi Francois,

> How does IBM have to change their integration thinking to ensure that current traditional integration products like MQ and IIB can smoothly move to the cloud era?

That’s a good question and so far I haven’t seen a lot of good answers. However, I have had many engagements where my client was virtualizing or moving MQ to the cloud and I’ve seen a number of common challenges. I’ll list a few of them here.

Many times with virtualization or cloud instances, the QMgrs are spun up with static names and this mitigates against them participating in a cluster. That in turn makes it expensive to achieve high connection density. What I have done for several clients is to modify the provisioning tools to query a configuration database. That way we can either assign or auto-generate unique QMgr names, spin up certificates, manage the cluster, etc. IBM has the beginnings of a central management framework with the web console and it may be the case that customers, vendors or even IBM enhance that with a configuration management system capable of interacting with the provisioning tools. Interestingly, the licensing of bundled components like WAS typically forbids use other than with the bundled products. The license anticipates you getting a copy of WAS with MQ or IIB and trying to use it to run business apps. It is possible that an exception is there or can be made if you extend WAS that comes with MQ specifically to manage the MQ network and not with business apps. IBM should make sure that licensing provision exists if it does not already, and also to make sure the documentation clearly explains the intention and allowed usage.

Rather than having many QMgrs, in some shops there’s a giant MQ hub “in the cloud” and everyone accesses it via MQ client. If the shop had previously had many local dedicated QMgrs, chances are the migration to a hub is based on license consolidation. While that is certainly one positive factor, issues with multi-tenancy are frequently overlooked. Security is pretty simple when there is one local QMgr and one or two service accounts but put 15 or 50 service accounts on a single QMgr and the number of access control list entries needed to ensure isolation rises exponentially. Similarly, isolation of access paths and shared resources like the DLQ becomes exponentially more difficult. For example, the tuning to fail a QMgr over quickly and that required to manage a great many concurrent transactions are mutually exclusive. Class of service issues are more difficult in a shared environment. This highlights a possible opportunity in that we do not currently partition best practice type of documentation according to network architecture. I’m not sure MQ can “smoothly move to the cloud era” unless our training and best practices become more architecture-aware. Unfortunately, the level of resource commitment to do this is probably prohibitive and especially so during the transition to CD/LTS delivery model and the documentation requirements that imposes.

Currently, MQ requires the existence of principals and groups to which security rules can be attached but it does not support security roles as an abstract entity. For example, there are a couple of roles that could be defined independent of the OS or of LDAP that are commonly required: Read-Only, Instrumentation, and Backup Operator. If we are going to provision things to the cloud it would be helpful if they showed up capable of being backed up, capable of interacting with the instrumentation and monitoring tools, and/or capable of being inquired on by desktop tools. I have many customers for whom provisioning accounts and groups are the primary blocker for virtualization of MQ. If MQ had a few standard roles out of the box (and perhaps the ability for users to create more) it breaks the hard dependency on the OS/LDAP and the deployed image could have CHLAUTH rules baked in that map authenticated connections to security roles.

We need altusr or MCAUSER equivalent for services. Currently you either run the service as mqm or else perform some black magic with setid or privilege delegation to get it to run as a different ID. The issues with privilege delegation are showstoppers in a lot of shops. Similarly, running services as mqm is problematic for other shops. It would be *much* simpler just to tell MQ “run this service as this user” or, if the previous suggestion were implemented, “run this service as this security role.” This is generally important for MQ and should have been done long ago, but when you move to the cloud and virtualization it is even more important to minimize the attack surface area. Reducing run-time privileges of services does that.

There are probably more of these but I suspect I’ve already gone well beyond anything IBM would budget for. In particular, IBM seems to focus on the code as product and not documentation as product, so although the most significant thing required for cloud is cloud-specific curriculum and docs, that’s the least likely suggestion to be implemented.



-- T.Rob

Hi T.Rob

Good to see your name again.  I think we initially started to talk on MQ in the middle 90s.  Anyway, VERY very good points you mention.

Just to comment on a few of the points:  MQ can really act as a disrupter in the cloud.  The technology is as solid as ever, it is still adding new features (after 24 years of being a distributed messaging engine giant) and as you said, it is starting to play a role in the cloud.  But yes, a few points are very painful.

Administration: In an elastic world with runtimes that stop and start because of the underlying infrastructure, you need an administration interface that can be made secure and fast for administrators.  One of the first questions asked by my customers in a multi-tenancy environment is (1) who admins?, and (2) how do I keep the admins out of my data?  And while small fast starting containers make it easy to keep single tenancy, it also means more granular administration that adds to time and cost.

Is there an easier way to consume MQ resources?  If you think of a queue as just a "service point", then maybe the IBM MQ designers must think about a more "API Connect" framework.  Give MQ resources its own section inside API Connect and you can gain all the governance and manementthat goes with it.  At the same time it will give you a "Cloud Service DNS" (something like your database) so that you do not need to worry where and how to connect to your MQ service.  Tie a bit of management and monitoring in and maybe it will work. 

I think the MQ Appliance suffer from some of these same problems.  One error log, one DLQ etc.  And maybe in the cloud and in the appliance you must be able to create many DLQs and assign queues to them so that every application can manage their own DLQ.

I so agree with you that MQ must have a few standard out of the box adminstration user classes with unique sequrity credentials....wow, will that not make life so much easier.  We implicitly trust MQ to deliver our ost valued messages, but boy-oh-boy, we do not trust the admins .... eish! (eish is South African for "face-palm, roll the eyes and shake your head in disbelief).

Looking at the documentation today for z and distributed, I think it is really good, but yes, putting cloud into the picture then it is really short.  MQ's nature of being plumbing and hidden from the average user also does not help in this regard.

I believe MQ will be a great tool overall in the cloud, but it really needs to roll out and start to do the job there as well.

Would love to receive comments and remarks. 

Hi.  Since this is my first time posting on the community, I'll take a moment to introduce myself.  I'm the Director for Application Integration at IBM and in this role I lead the Offering Management, Engineering and Design teams for IIB and App Connect.

This is a GREAT, GREAT question.  We at IBM have been investing a tremendous amount over the past two years in modernizing our integration portfolio to tackle the challenges of cloud.  I can elaborate a lot on this topic, but let me try to keep it concise here and leave room for follow-up questions:

• Integration Bus on Cloud ... this has become a very successful offering over the past 18 months since it was launched.  Customer had immediate interest as they could immediately take all of their existing skills in building enterprise-grade integrations and have IBM manage the environment where it is running.
  
  
• App Connect ... last year we brought a new experience to market to fit the iSaaS space (iSaaS being "integration software as a service").  This basically requires 0 training to get started, is freely available to anyone up to a certain level of use, and covers multiple integration patterns in this native cloud, multi-tenant environment.  We add new function to App Connect on about a weekly basis.  Among it's strengths is a growing list of new cloud connectors.  Check it out here ... appconnect.ibmcloud.com
  
  
• Application Integration Suite ... packages IIB and App Connect, along with our API management technology (API Connect) and is available as a hosted service, on premise, or a 'hybrid license' which gives access to both deployment options.  This is a very flexible licensing model that appeals to organizations that are shifting their middleware into the cloud.
  
  
• Coming soon ... we have a lot of new cloud technology planned to be brought to market.  Among those objectives are a few key ones...
• bringing IIB and App Connect together into one offering
• even lighter-weight, microservice friendly runtime (what we call Project Firestorm)
• new cognitive technologies to simplify the integration build process

If anyone is looking for more information on this, I'd be very happy to connect more directly.  Please shoot me a note at tcurcio@us.ibm.com.

Hi Tony, and thank you for engaging in this platform. 

Having a look at integration patterns on the cloud I think it is clear that small self contained microservices will play a very important role.  Throw technologies like Kubernetes and elastic patterns into the picture and the monitoring and management can become very hairy to say the least.  It also has the potential to create a runaway pattern with the end result that somebody must maybe pay a huge amount of $ for usage.  The traditional on-prem monitring might not work in certain cases.  Do you see any enhancements in this area?

Regarding the monitoring, IBM has done some good work over the last year with respect to Product Insights.  It provides a mechanism for any instance of our middleware products to report usage into a centralized location (a service in IBM's cloud).  That affords organizations to have a cross-organization view of the products, key metrics and logs from technologies like WAS, IIB, MQ, DB2, and others.  And best part is that it's free.  :)

The pattern you describe where we will want to collect logs from a series of "integration microservices" is not very different - at least not from the decentralized nature of the components.  As we move forward, you will see that pattern get used repeatedly.  In fact, App Connect .... which is constructed entirely around a set of microservices ... already provides a similar consolidated logging capability in working with our logmet service (which is also at the heart of Product Insights).

 As for pricing, we provide two types of metrics ... capacity based (let me run this on N cores on prem or on cloud) and usage based (I'd like to pay for only the number of flows/apis/etc... that I use).  These provide customers flexibility to start small and grow as their needs expand.  Of course, we are always planning the next thing, so if anyone is reading this and would like to provide additional feedback/considerations/etc... please share that opinion.

And finally ... on a lot of moving parts.  One of the benefits of our managed service is that you reduce a number of your moving parts immediately.  We take care of the infrastructure and most of the admin (I say most, because customers still need to manage who is a user, will want to review for health of their overall solution, etc...).  So, that is immediately a reduction in moving parts.  One of the reasons I'm fond of our Application Integration Suite offering is that it further reduces the number of moving parts from a licensing standpoint.  You get one item on your bill, and that includes our application integration engine, all of our cloud connectivity, and the api management features.

Another question: Pricing.  Due to the nature of integration there is often MANY moving parts, and it does make it difficult to price a cloud integration solution.  Any advice / tools available on this topic?

Thanks for the reply Tony! Since this is the WSMQ Family list I trust it is fair play to make the observation that the letters “M” and “Q” did not appear adjacent to one another in your description of how IBM is approaching cloud. Is that because MQ is not in your portfolio as offering manager or does that say something about MQ’s declining role in the strategy for cloud?

-- T.Rob

Hi T.Rob....

I saw the "MQ and IIB" reference above so took some liberties since I own the IIB portion of the portfolio ;)

MQ on Cloud .... a lot of work going on there, but since it's not mine, I should avoid being too specific on some of the secret sauce.  I'll give Leif Davidsen a heads up and ask him to weigh in here.

---Tony