Co-Author - Carsten Boernert
This blog is part of a series. For the whole series list see here
When deploying IBM App Connect Enterprise integrations in containers there are options around how we deploy the runtime, the configurations, the integration flows and environment specific data. All the movable parts can either be “included in the image”, “configured at deploy time” or a mixture of the two. In this article we will explore these different deployment options, the benefits of each and considerations for taking each approach.
What changes when we deploy IBM App Connect Enterprise in containers?
From IBM App Connect v11 the integration server can be deployed on its own, without the need for an integration node to look after it. This is most relevant in a container scenario where the container platform itself takes on the management of the individual integration servers. Additionally, all the IBM App Connect Enterprise configuration required is defined alongside the container providing a more portable deployment of integrations. Container based deployment has many notable differences to a traditional deployment, such as:
- Deployment isolation: In container deployments we see groups of integrations being independently deployed into their own container runtime as opposed to a more traditional singular centralized ‘broker’ deployment. This allows for greater isolation, protecting each deployment from the other integrations.
- Product runtime versioning: Since the container image also includes the product runtime this provides an opportunity to, for example, run different versions of IBM App Connect Enterprise in different containers allowing each integration to independently choose when and how to upgrade. Previously in a shared server environment all integrations were running on the very same version of the product and had to be upgraded at the same time, with costly regression testing, and greater risk during the single upgrade.
- Memory and CPU: Container platforms enable explicit/declarative ways to specify how much CPU and memory each (IBM App Connect Enterprise) container requires, ensuring the specific needs of the integrations can be taken into account.
- Dedicated Operating System Resources: With VM/server-based IBM App Connect Enterprise deployments there are a few areas where resources just could not easily be dedicated and managed, meaning unrelated workloads could potentially impact or have adverse effects on each other. An example of this would be the integration node level listener or the maximum number of threads or processes on the VM/server.
- Load balancing: In a traditional VM/server-based deployment, load balancing across VM replicas has to be explicitly set up and maintained. In containers this is replaced with the dynamic routing which is automatically set up and managed by the container platform as containers are deployed.
- TLS termination: In containers this would potentially be handled and configured by the platform (i.e. outside of the container) which in traditional VM/server-based deployments the configuration would be in a configurable service. This can be managed through a ‘Service Mesh’ like Istio which you can read about in the O’Reilly book ‘Istio Explained’ - https://www.ibm.com/downloads/cas/XWN1WV9Q
- Declarative Configuration: The majority of the configuration is now declaratively defined within a configuration file (server.conf.yaml) rather than requiring post deployment IBM App Connect Enterprise commands to be run on the node and server. This is linked with container-based objects like Kubernetes ConfigMap that are read on start-up, ensuring that the integration server’s state is not changed at runtime. This means configuration is more portable and repeatable as the documented YAML defines the true state of a redeployed container.
What is a container image and what to include in it?
A container image is a read-only template which is used to create instances of containers. Anything not included in the image can then be sourced during the creation of the container. It is not part of the deployed image but will be available in the running container.
Therefore, when deploying IBM App Connect Enterprise into container environments there is a decision to be made on how much of the configuration (including external libraries, drivers etc.), flows and environment settings should be included as part of the image being deployed.
Figure 1 – Shows what is included in each container image type
Sometimes the following terms associated with the scale shown in Figure 1 are:
- ‘baked’ (furthest right column) as in everything is baked into the image,
- ‘fried’ (further left column) where everything is configured as it is needed.
Most organizations will fall somewhere between these two extremes, but most commonly we see people using the image types C and D from Figure 1.
All options represented in the diagram are valid ways of building and deploying integration containers. Generally, the decision as to which approach to take relates to secondary considerations such as image management, skills within a team, non-functional requirements (testing, security) and management of the live environment.
Using pre-built container images
IBM provides a pre-built container image for IBM App Connect Enterprise in the IBM Entitled Registry. This image contains pre-integrated database drivers, and the MQ Client. Note that this pre-built image can only be used in conjunction with the App Connect Operator. On deployment of the image, you can specify the location of a remote BAR file and links to configuration information, and the container will draw those in during start up. Used in this way it is therefore an example of Image type B in Figure 1. It is also possible to build new images based on the pre-built image, adding server configuration, a BAR file, or even environment configuration in order to create image types C, D, or E.
Whilst the pre-built image covers most scenarios, there will always be circumstances where teams would prefer to create a completely custom image. A sample is provided on github for this purpose. However, there are many advantages to using the pre-built image with the App Connect Operator as described in the following articles:
Images and Hierarchies
New images can be based on existing images.
Figure 2 – Basing Images on other images
Image 1 could be deployed, adding the BAR file and server configuration at deployment time. Alternatively, further images could be built by layering configuration (e.g. server.conf.yaml) and code (e.g. the BAR file) on top of the first image as in Image 2 in Figure 2. In this case we would only have to add the environment specifics at deployment time.
We could go even further and create an image for each environment that the integration will be deployed to as in Image 3 in Figure 2. In this case we could simply deploy the image and it would automatically contain all the environment specific information required.
These images are hierarchical, relying on each other, meaning a level of management is needed. Part of this management is the responsibility of ownership of the images, from the template owner through to the application teams owning environment specific images and containers.
Having a single container with everything deployed into it doesn’t mean just one image to manage. All three images shown in Figure 2 will need to be managed and consideration given when making changes, i.e. how a change in Image 2 might impact the top level of images (Image 3) with environment specific configurations.
The illustration below provides an example of an image hierarchy where images have been created down to the level of each environment. Let’s look at the implications of this;
Figure 3 – Example image hierarchy where environment specific images have been created
We can see that making a change to an upstream image (such as “Rest Flow”) potentially affects many downstream images and could cause a cascade of image updates to be required.
On the one hand this might be considered good for governance as it enforces that all images share a known base. On the flip side, small changes to images at the base of the hierarchy could result in sweeping changes, which detracts from the isolation and decoupling we’re trying to achieve in more fine-grained deployment.
Image template configurations
Certain groups of integrations will share common aspects. The IBM App Connect image from the IBM Cloud Container Registry provides a good base template containing database drivers and the MQ Client. Further templates can be built on top of that to include any additional elements and can be used as ‘Enterprise Templates’ to be managed and maintained by the organization for all future integrations of a given type.
Such elements found in Templates, are well-suited to be “included in the image” rather than “configured at deploy time”. Examples might include the addition of database drivers, certain client libraries and agents, and maybe even standardized sub-flows for things such as error handling.
The core App Connect Enterprise runtime configuration is detailed in the server configuration YAML file (server.conf.yaml). Certain settings in that server configuration can also be included as part of an image template, to implement organization wide standards and defaults. Examples might include administrative role mapping, configuration for offloading logs to an external system and more.
Figure 4 - Image Templating
IBM App Connect Enterprise specific considerations
This section looks at a few key considerations for IBM App Connect Enterprise when deciding what to include within the container image.
IBM App Connect Enterprise is a rich runtime with a lot of additional configurations that can be deployed to describe the integration server, the container itself, the integration flow and environment variables for each of these. This exacerbates the potential for environmental drift when configuring entirely at deploy time and the complexity of the build when configuring during the build of the image.
Implementing integrations inherently means that there are more configurations compared to traditional application development. For example, for each integrated system there are connectivity settings, clients, drivers, credentials etc. Furthermore, getting those configurations exactly right is critically important to the functioning of the integration application.
In Figure 1, we see different approaches for how much of these different types of configurations can be included in the image.
Choosing an image strategy more towards the left of Figure 1 can lead to a very large number of live configurations to manage when configuring them all at deploy time. As we move toward the right of Figure 1, we include more of these configurations within the boundaries of the image. This reduces the potential for environmental drift and enables testing during image build time to catch errors and issues before deployment into each environment.
However, the more configuration you build into the image, the more images you will have to manage, to the point where in the furthest right column of Figure 1, you would have a different image for each application for each environment.
Observations from the field
As organizations begin this journey, we are often asked how other customers are performing their own container deployments.
All the approaches shown in Figure 1, from using the certified container and configuring everything on top at deploy time, through each of the five approaches to having environment specific images, are valid deployment styles.
Some of the customers we have worked with have been concerned around environmental drift and as such have included as much configuration into the image as possible, up to and including the BAR file. These customers tend to configure environmental specifics at deploy time in a bid to reduce the number of images they need to manage.
Other customers have gone a step further and created images for each environment, however we have not seen enough customers doing this to see this as a general trend.
Given the number of styles available and the number of implementation decisions to be made, there is no reason why the deployment style cannot be refined as the organization better understands the implications.
Thoughts on choosing an image deployment strategy
As organizations transition more into container-based deployment models, there is a significant change in how we approach the build and deployment of IBM App Connect integrations.
There is no ‘one size fits all’ when deciding which style of container-based deployment to adopt. It is important to understand how these changes will impact your organization, its needs, and ways of working. You need to explore questions such as the following:
- Categorizing your integrations: How many common integration “patterns” are present? This may indicate how many different image templates you may need.
- Credential Management: How will credentials be managed and accessed either during build or deploy time?
- Tooling: Do you have the right tools and skills in your organization to implement the building and deploying of images, configurations, and related artefacts? How do you manage and access artefacts?
Stay tuned to this series, for some practical examples of including configuration in images and building image hierarchies.
Acknowledgement and thanks to Kim Clark, Rob Convery and Ben Thompson for providing valuable input to this article.