Special thanks to: Trevor Lobban, Daniel Robinson, Kim Clark, Ben Thompson, Gaza Geleji, Amar Shah
Introduction
This article details how certificates are configured within IBM App Connect and covers version 13.x on OpenShift. This is intended for App Connect administrators, operations teams and security teams with a focus on App Connect. The article highlights key areas of certificate management including the creation and renewal of certificates.
----
IBM App Connect on OpenShift
Figure 1 – IBM App Connect on OpenShift Objects
IBM App Connect is application integration software that connects applications and data sources which enables message parsing, transformation, serialization and aggregation across various protocols.
The OpenShift Operator looks after multiple objects:
- Integration Runtime: An isolated runtime environment for a set of deployed message flows and resources which runs as a unique process in a separate address space
- Dashboard: A user interface which shows the deployed Integration Runtime via the containers exposed REST Admin endpoint
- Configuration: An App Connect specific object which has multiple types, each type is placed in different areas of the container. For example, an odbc.ini, keystore, truststore, MQ Policy project.
Configurations are base64 encoded inside a Kubernetes secret.
There is an additional decision point for OpenShift around whether to deploy with Cloud Pak for Integration (CP4I) or directly onto OpenShift. App Connect deployed with CP4I requires Foundational Services, Platform UI and the App Connect Dashboard to be deployed.
This is important for the certificate discussion because it means that certain routes for App Connect are redirected via the Platform UI or the App Connect Dashboard.
----
Defining App Connect Certificates
Figure 2 - App Connect Certificate Definitions
There are two primary ways of defining certificates in App Connect Enterprise. Generally certificates are defined in the Integration Runtime via a server configuration yaml file (server.conf.yaml), however, certificates can also be referenced inside an App Connect Enterprise message flow.
----
Certificates defined in the server.conf.yaml
|
Type
|
Purpose
|
Certificate Type
|
|
BrokerRegistry
|
Top level configuration that - when set – defines the default java SSL configuration across the integration server
|
JKS (Default) or P12
|
|
JVM
(Resource Manager)
|
If no BrokerRegistry SSL is set, the JVM becomes the defacto top level object. The defined java SSL configuration becomes the default inherited by other resources.
This is used by Java based applications, such as the Java Compute Node.
|
JKS (Default) or P12
|
|
HTTPSConnector
(Resource Manager)
|
The HTTPS Connector is used for secure HTTP traffic into the integration server. It uses OpenSSL which enables the use of the PEM format. If JVM SSL is configured, the certificate is configured from these certificates using c++ libraries.
|
JKS (Default) or P12 and PEM
|
|
OpenTelemetry
(OpenTelemetryManager)
|
Used for OpenTelemetry communication, also inherits from the JVM if not explicitly set.
|
P12 or JKS
|
|
ELKConnections
|
Used to enable to connections to ElasticSearch endpoints. Inherits from the JVM if not explicitly set.
|
JKS
|
|
REST Admin
|
For REST administration ingress to the integration server, a certificate must be provided which is utilised using the Java language SSL functions.
|
P12 or PEM
|
|
LDAP
|
LDAP uses the certificates defined in the REST Admin section of the server.conf.yaml.
|
P12 or PEM
|
|
Other Third Party Certificates
|
Other certificates might be needed for integration with downstream services. In some cases, pre-formed certificates are uploaded to a directory (e.g., MQ Client certs), in others, .crt and .key files can be used with a startup script that builds a specific format at deploy time e.g., SAP PSE certificates that need the SAP binaries to generate.
|
Various e.g., PSE for SAP
|
It’s important to note that the server.conf.yaml does have a hierarchy. The BrokerRegistry when set is used by default by all nodes, if the JVM is set the other socker elements use the JVM otherwise the BrokerRegistry. If the individual properties are set, their configuration is used.
----
Certificates defined in Message Flows
App Connect has grown over the years adding new features, connectors and adapting to new messaging paradigms and integration patterns. As such, there are some considerations for certificate usage when developing message flows.
At a high level, Message Flows support one of the following methods for defining certificates:
- Utilisation of the Node.js libraries with some form of alternate authentication with the services being connected to. This is most common in Discovery Nodes e.g., GitHub, GitLab, Anaplan.
Certificates are provided out of the box, and services being connected to are trusted if they are signed by an accepted global signer. Mutual authentication is not possible.
These nodes typically rely on additional forms of authentication e.g., BASIC, API Key, OAuth(2), SAML
- Utilisation of certificates in the server.conf.yaml. This typically enables mutual authentication;
- JVM – Java Databases (JDBC), Connect:Direct, JMS, TCPIP
- HTTPSConnector – SOAP, HTTP Input, REST
- BrokerRegistry – Email, File, HTTP Request, SAP Input / Reply
- ODBC.ini file is used to configure certificates for Oracle Database connections
- Utilisation of uploaded keys and certs to the ACE pod which are then referenced in Policy Files or the Node itself. These do typically permit mutual authentication. This includes;
- Kafka
- SAP Request (PSE)
- FTE
- MQ (MQSSLKEYR which references a PEM in a credential)
- CICS
- Utilisation of a Security Profile or Security PEP Node (x.509 certificate) allows for keystore and password locations to be set. But these can only be used with specific message flow nodes.
Note: that when certificates are defined directly in message nodes, renewal can be completed with a simple application rollout of a new version. When combined with normal Kubernetes procedures this can be non-disruptive.
----
Managing App Connect Certificates
Out of the box, App Connect doesn’t have certificates defined, they must be built into the image or injected either as a secret or as an ACE Configuration.
There are three primary methods for managing certificates to be used by App Connect in an OpenShift Platform:
Figure 3 - Three primary methods of managing certificates
----
1. Certificates uploaded as secrets into OpenShift and referenced in the custom resource
Certificates can be generated outside of OpenShift and then imported into OpenShift as a secret to be used by App Connect. This does have an overhead of management by administrators who need to push the changes into OpenShift even if automated.
Certificates should be signed by a trusted certificate authority (CA), although the CA signer certificate does not need adding into OpenShift.
This approach is more suited when a third party provides certificates for you and there is a desire to add certificates at deploy time rather than include them in an image.
----
2. Certificates generated and managed as secrets by a certificate-manager issuer object within OpenShift, which are then referenced in the custom resource
A single certificate can be uploaded into OpenShift - a certificate authority (CA) or Intermediary (ICA) signer - which can be used as a Cert Manager issuer object.
Cert Manager is a Kubernetes deployment which manages and generates certificate objects, observing expiration dates and automatically generating and updating certificates.
The issuer object allows users and services to generate certificate objects that are automatically signed by the issuer meaning administrators must only manage the CA secret and not the individual secrets.
Cert Manager can also integrate with third party systems, making calls to have the certificates it managed signed by an issuer that is external to the cluster, for example, Venafi and Letsencrypt.
This approach is more suited when the platform is managed by a certificate manager with a single certificate used as the certificate issuer for applications and services on the cluster. This approach pushes certificate changes reducing administrative overhead either through automatic or manually triggered renewal of certificates.
----
3. Certificates built into the custom container / image
Certificates can be embedded into the container image at build time, for example, by copying them into a specific location in a Dockerfile (which is what Kubernetes secrets do under the covers).
This process can be completely agnostic to the container platform being used e.g., OpenShift and creates immutable images, often with environment specific configuration embedded in the container image,
The container image with certificates embedded can be stored in a repository and deployed directly to the environment. This method relies on certificate management at image build time and as such, certificate injection needs to be embedded into an automation pipeline or manual process.
This is more suited where the standard deployment approach is to have environment specific container images.
----
Benefits of fine-grained integration deployment
Figure 4 - Coarse versus fine-grained deployments
A benefit of fine-grained deployment of integrations, more commonly seen in container-based deployment, is that we can isolate the certificates required by a specific integration. Compare this to a coarse-grained deployment where there is a risk that certificates could be used by integrations for which they were not intended, either accidently or indeed maliciously.
An additional natural consequence of fine-grained deployment is that certificates can be updated for a specific integration in isolation, with minimal/no disruption of others.