App Connect

 View Only

Installing the IBM App Connect Operator on Amazon EKS

By Joel Gomez posted Tue March 08, 2022 02:44 PM

  
This article explains how to install the latest version of the IBM App Connect Operator on an Amazon EKS cluster, and deploy an integration using it. It is part of a broader series on moving to IBM App Connect on containers.

Most of our scenarios in the series so far have assumed that your Kubernetes environment is Red Hat OpenShift. This reduced the steps in our examples significantly as OpenShift pre-integrates a huge number of commonly used capabilities (200+) and performs a number of common steps automatically. However, we recognize that not all enterprises have OpenShift available to them, and for that reason, IBM App Connect is designed and supported to run on any major Kubernetes distribution.

In this example we’re going to see what additional steps would be required to perform a deployment on Amazon Elastic Kubernetes Service (EKS) using the IBM App Connect Operator. In order to keep this post to a reasonable length, we have assumed that the reader has some familiarity with Amazon EKS already, and hence the reason they have taken an interest in this post. As such we will refer the reader to EKS documentation where necessary rather than repeat it here.

We will set up the EKS environment with the necessary capabilities that would have already been present in OpenShift, then we’ll perform an actual deployment of an integration.

To prepare the environment we will need to do the following extra steps on Amazon EKS compared to OpenShift:

  • Install a certificate manager
  • Install the Operator Lifecycle Manager (OLM)
  • Create the Apache CouchDB operator catalogue
  • Install the App Connect Operator manually
    • Create an Operator Group for the IBM App Connect Operator
    • Create the IBM App Connect operator catalogue
    • Create the IBM App Connect operator subscription
    • Optional: Create App Connect Dashboard
  • Create an NGINX Ingress

We only need to do the above preparation once. However, for each integration that needs to be exposed on the network outside the Kubernetes cluster we will need to:

  • Create an ingress definition for the integration

The following diagram provides a visual representation.

Differences between EKS and OCP


In order to follow these instructions, you will need to have the following tools already installed on your local workstation.


Everything we will describe in this blog post is also present (in greater detail) in the IBM App Connect documentation here.

Configure your AWS and EKS command line

Throughout this post we will be remotely connecting to Amazon EKS from our local command line. We will set up a local configuration such that we don’t need to enter details such as credentials on each aws or kubctl command.

Configure your AWS CLI


First, we need to configure basic settings (credentials, region etc.) that the AWS Command Line Interface (AWS CLI) uses to interact with AWS.

$ aws configure
AWS Access Key ID [****************J74P]: 
AWS Secret Access Key [****************D12u]:
Default region name [us-east-1]:
Default output format [json]: 

There is more information on configuration in the AWS CLI documentation.


Create/update your EKS cluster configuration


Next we need to create a kubeconfig file that stores credentials for our Kubernetes cluster on EKS.

$ aws eks update-kubeconfig --name <cluster-name>

There is information on creating a kubeconfig in the Amazon EKS documentation.

Install Cert-Manager in your EKS cluster


Any communication to components that needs to be secured using TLS (such as HTTPS) will require certificates.

The operator installation requires that you install the Kubernetes certificate management controller (cert-manager) in your cluster to generate and manage the TLS certificates that are required for internal communication, and exposure of web user interfaces. This will also be required later by the integrations we deploy if they expose for example HTTPS APIs.

“Cert-manager” is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. Cert-manager is included in OpenShift, but in EKS it has to be explicitly installed.

These instructions were correct at the time of writing, but feel free to check the documentation for any more recent updates such as the version number of cert-manager.

Install cert-manager using the following command:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

You can verify the cert-manager installation using:

$ kubectl get pods --namespace cert-manager

We also need to perform a small patch to cert-manager to ensure any auto-generated secrets that store certificates are automatically removed when there are no longer any “owner references”. We do this with the following command:

kubectl patch deployment \
cert-manager \
--namespace cert-manager \
--type='json' \
-p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
"--v=2",
"--cluster-resource-namespace=$(POD_NAMESPACE)",
"--leader-election-namespace=kube-system",
"--enable-certificate-owner-ref"
]}]'

You can use the following command to confirm patching worked as expected:

kubectl get deployment cert-manager --namespace cert-manager -o jsonpath='{.spec.template.spec.containers[0].args}'

You will get a response like the following showing the values you just added with the previous patch command:

[--v=2 --cluster-resource-namespace=$(POD_NAMESPACE) --leader-election-namespace=kube-system]


Install the Operator Lifecycle Manager (OLM) in your EKS cluster


The Operator Lifecycle Manager (OLM) is a fundamental part of the Operator Framework. It is a Kubernetes application that looks after your operators (such as the IBM App Connect Operator). It makes operators discoverable, ensures that installed operators are kept up to date, manages their dependencies and underpins many of the core aspects of the operators’ functionality. It is installed in OpenShift by default, but this is typically not so in most other Kubernetes distributions such as EKS.


Install the Operator SDK

The Operator SDK is primarily for building your own operators, but it also happens to include the command needed to install the OLM capability on your Kubernetes cluster. Follow the instructions to install it here:

https://sdk.operatorframework.io


Install the OLM

Use the SDK to install the OLM

$ operator-sdk olm install

This command completes with a list of the resources that it has created to verify the installation.

You can also verify the OLM installation with

$ kubectl get crds

The output you should expect to see is documented here along with full details on this step.

Add Apache CouchDB to the operator catalog

The IBM App Connect operator has a broader use than just deploying IBM App Connect Enterprise runtimes. It also provides the capabilities for creation of integrations using the App Connect Designer tooling. Due to this known dependency CouchDB will need to be available in the catalog for the IBM App Connect Operator installation to succeed.

The OLM installed in the last step holds a catalog of available operators. In our OpenShift based scenarios, CouchDB was available because we had populated the OLM with the whole IBM Catalogue, since this is easy to do on OpenShift. In this EKS based example we’ve chosen to take a more minimalist approach and we will just populate the catalogue with the Operators that we need.
To add a new entry in that catalogue you supply a CatalogSource object. For our CouchDB operator the catalogue source is:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: couchdb-operator-catalog
  namespace: olm
spec:
  displayName: "IBM CouchDB Operator Catalog"
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/couchdb-operator-catalog:latest
  updateStrategy:
    registryPoll:
      interval: 45m

Place the above into a file named couchdb-catalog-source.yaml

Now run the following command to add it to the OLM catalog.

$ kubectl apply -f couchdb-catalog-source.yaml -n olm

You can verify the operator is available in the catalog with:

$ kubectl get CatalogSources couchdb-operator-catalog -n olm


Install the IBM App Connect Operator


Installation of the App Connect Operator itself involves three steps:

  • Create an OperatorGroup
  • Add App Connect to the operator catalog
  • Install the operator by creating a Subscription


In our previous examples OpenShift’s OperatorHub made reasonable assumptions and turned this into a single, relatively straightforward step via the web console. In EKS we have to do each step manually.


Create an OperatorGroup for the IBM App Connect Operator


The OLM runs with high levels of privilege within Kubernetes, and can grant permissions to operators that it deploys. An OperatorGroup provides a mechanism for controlling what permissions are granted by OLM, and in which namespaces. If Kubernetes administrators have any security concerns relating to OLM, this is a critically important area to explore.

A single OperatorGroup must be created in each namespace that will contain operators. The namespace for our operator will be ace-demo and we need to create an OperatorGroup within that.

Create the app-connect namespace for our operator:

$ kubectl create namespace ace-demo


Create an OperatorGroup definition by creating a file named appconn-operator-group.yaml with the following content:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: app-connect-operator-group
namespace: ace-demo
spec:
targetNamespaces: -ace-demo


It is of course important that the namespace above matches with the namespace into which we later try to deploy our integration into. We have chosen the same namespace as we have been using for our earlier scenarios.

In a production installation you might choose to have the operator namespace different to the target namespace, and create a separate ServiceAccount to reduce the permissions made available to the IntegrationServer. However, a single namespace is sufficient for our simple example.

Now create the OperatorGroup.

$ kubectl apply -f appconn-operator-group.yaml


Add IBM App Connect to the operator catalog


OLM doesn’t yet know the IBM App Connect operator exists, or where to get it from. We need to create the entry in the OLM catalog by creating a CatalogSource object.

Create a file named appconn-catalog-source.yaml with the following content:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-appconnect-catalog
  namespace: olm
spec:
  displayName: "IBM App Connect Operator Catalog k8S"
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/appconnect-operator-catalog-k8s
  updateStrategy:
    registryPoll:
      interval: 45m


Note that the namespace for the CatalogSource is always olm.

Now add it to the OLM catalog:

$ kubectl apply -f appconn-catalog-source.yaml -n olm


You can verify the catalog entry using:

$ kubectl get CatalogSources ibm-appconnect-catalog -n olm


Install the operator using a “Subscription”


For this final step, you might have been expecting some simple command to install an operator. However, as I’m sure you’ve gathered by now, Kubernetes works “declaratively”, and doesn’t like being told what to do! It likes you to “declare” what you want, then it works out how, and indeed when to do it.

Installing an operator on the OLM follows this same pattern. You declare to the OLM that you would like it to “subscribe” to a particular operator in the catalog. It then takes on the task of downloading the current version of it and instantiating it. There are many advantages to this declarative approach. For example, we can declare that we want to follow a particular “channel” of updates to the operator; for example, we might want every update, or just stable releases and so on. We can also declare whether or not it should update the operator automatically or not.

To create a subscription to the IBM App Connect Operator, create a file named appconn-sub.yaml with the following contents:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ibm-appconnect
  namespace: ace-demo
spec:
  channel: v3.1
  name: ibm-appconnect
  source: ibm-appconnect-catalog
  sourceNamespace: olm


Once again, pay close attention to the namespaces.  The metadata.namespace must be the one we will be deploying our integration to. The spec.sourceNamespace is always olm.

You will notice that we will be subscribed to the “v3.1” channel. This means we’re subscribing to a specific version of the operator which was correct at the time of writing. Do check the current documentation to see if a different version is now recommended.

Add the subscription:

$ kubectl apply -f appconn-sub.yaml


To verify that the operator has installed:

$ kubectl get csv

NAMESPACE        NAME                                        DISPLAY                                             VERSION     REPLACES                                 PHASE

ace                         couchdb-operator.v2.2.1     Operator for Apache CouchDB     2.2.1             couchdb-operator.v2.2.0     Succeeded

ace                         ibm-appconnect.v3.0.0        IBM App Connect                             3.0.0                                                                 Succeeded



The important field is the PHASE, which should have a status of Succeeded. Note that you may need to specify a namespace, or perhaps use the –all-namespaces option.

Creating an Ingress for your EKS cluster

This step is arguably optional since the operator’s installation isn’t actually dependent on it. However, at this point, your cluster is unable to receive requests from the outside world.

It would be rare to find an “integration” that didn’t need to receive some form of request from beyond the EKS cluster. As a case in point, our example in the next post will expose itself via HTTP. To permit HTTP requests from beyond the cluster, we will therefore require what is known in Kubernetes as an “ingress gateway” in order to make the integration callable over the public internet. Again, this is something that is implemented in OpenShift cluster by default, but in EKS you need to create this explicitly.

Using the following command to create a basic ingress gateway based on the NGINX runtime. There are many other ways to create an Ingress depending on your requirements.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/aws/deploy.yaml


It is beyond the scope of this post to explore the deployment file for this ingress gateway, but there is a good blog post discussing it here.

Optional: Create an instance of the App Connect Dashboard

As we’ve previously shown in Scenario 2a, you can deploy and administer App Connect Integration Servers completely from the command line, and that is all we need for this example. However, there is a user interface available, known as the App Connect Dashboard. If you want to use this user interface, you should at this point explicitly create an instance of the dashboard. The instructions for this would be the same as those in Scenario 2b , but with one addition. You must create an Ingress such that the dashboard UI is available beyond the EKS cluster as documented here.

Deploy your integration

So, that’s all the one-time preparation done and we can move on to deploying an actual integration.

We can now deploy in exactly the same way as we did in our previous scenarios. However, note, there will be one additional step that we didn’t have to do in Scenario 2a; the creation of an Ingress definition.

We’ll provide the absolute minimum instructions here, but feel free to go back to the Scenario 2a blog post if you want a refresh on the concepts and the reasoning behind each step. The main difference  is that instead of working with OpenShift, using the “oc” command, we’ll instead be using the essentially identical Kubernetes command kubectl.

  1. Create the barauth configuration object

    Create a text file named github-barauth.yaml with the below contents.

    apiVersion: appconnect.ibm.com/v1beta1
    kind: Configuration
    metadata:
      name: github-barauth
      namespace: ace-demo
    spec:
      data: eyJhdXRoVHlwZSI6IkJBU0lDX0FVVEgiLCJjcmVkZW50aWFscyI6eyJ1c2VybmFtZSI6IiIsInBhc3N3b3JkIjoiIn19Cgo=   description: authentication for github   type: barauth

    You may remember from Scenario 2a that the data section is the base64 encoded string for an “empty” password since we are pulling the bar file from public github.

    Apply the barauth configuration definition to Kubernetes

    $ kubectlapply -f github-barauth.yaml
  2. Create an Integration Server for the simple integration

    Deploy an IBM App Connect certified container with a link to our simple BAR by creating a file named http-echo-service.yaml with the following contents.

    apiVersion: appconnect.ibm.com/v1beta1
    kind: IntegrationServer
    metadata:
      name: http-echo-service
      namespace: ace-demo
    spec:
      adminServerSecure: false
      barURL: >-
        https://github.com/amarIBM/hello-world/raw/master/HttpEchoApp.bar
      configurations:
        - github-barauth
      createDashboardUsers: true
      designerFlowsOperationMode: disabled
      enableMetrics: true
      license:
        accept: true
        license: L-KSBM-C37J2R
        use: AppConnectEnterpriseProduction
      pod:
        containers:
          runtime:
            resources:
              limits:
                cpu: 300m
                memory: 350Mi
              requests:
                cpu: 300m
                memory: 300Mi
      replicas: 1
      router:
        timeout: 120s
      service:
        endpointType: http
      version: '12.0'

    Note that strictly speaking you could set createDashboardUsers to false if you know you are not creating an instance of the dashboard.

    Create Integration Server with the following command:

    $ kubectlapply -f http-echo-service.yaml
    You should receive the confirmation:

    integrationserver.appconnect.ibm.com/http-echo-service created
  3. Create ingress object

    This is the final extra step that you have to do on EKS compared to OpenShift. Our simple integration receives requests over HTTP and these may come from the public internet. To enable a container to receive requests from outside of the EKS cluster you will need to create a definition on the ingress gateway we created earlier.

    Create a definition file named is-ingress.yaml with the following contents, taking care to replace <external-hostname> with the correct value based on the instructions that follow:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: http-echo-service-ingress
      namespace: ace-demo
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: <external-hostname>
        http:
          paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                 name: http-echo-service-is
                 port:
                   number: 7800

    Here are some brief explanations for some of the relevant fields in the ingress definition file:

      • metadata.name – This can be any name, but you should use a naming convention that helps to identify the relationships with the Integration Server, i.e. <deployment-name>-ingress.
      • metadata.namespace – The namespace in which you deployed the Integration Server.
      • spec.rules[].http.paths[].backend.service.name – This value must match the service created when the Integration Server was deployed. The name is going to be the name of the Integration Server instance followed by “-is”, something like <deployment-name>-is. You can use command kubectl get services -n $NAMESPACE to confirm the name.

    The subtlty here is that you must work out a suitable value for <external-hostname> in the  spec.rules[].host field. This will ultimately be the external address external consumers will use to call the integration. What you use for this value depends on your DNS setup. The most important thing is that it has a prefix that allows the ingress gateway to uniquely route traffic to your integration serverpod. A simple example might be something like:

    <IntegrationServerDeploymentName>.yourdomain.com
    which in our case would be:

    http-echo-service.yourdomain.com

    However, it’s worth considering whether this is unique enough. Remember, the ingress gateway we created earlier is cluster wide, so what if you had another deployment of the same integration in another namespace?
    If you look back at Scenario 2a you’ll see that, this ingress definition was created for us automatically by the ACE Operator. The operator only currently does this when running in OpenShift, because it knows there is always an Ingress gateway present. (Note that ingress definitions in OpenShift are called “Routes”).

    The prefix that the Operator created automatically in Scenario 2a was constructed from three things.
                 1. The integration server deployment name “http-echo-service
                 2. The protocol “http” (we could be exposing the same service over HTTP or HTTPS)
                 3. The namespace “ace-demo

    So the prefix ended up being “http-echo-service-http-ace-demo”. A little complicated perhaps, but with a reasonable guarantee of uniqueness across the cluster.

    So, a more unique value for the full <external-hostname> might be something like:

    http-echo-service-http-ace-demo.yourdomain.com

    Do you have a DNS configured?

    If you have an already established EKS environment in your organisation, it is likely that you also have a DNS configured. However, if you are using a temporary test environment, it is possible that you do not. In this case you may need to use a wildcard DNS service such nip.io. These allow you to use any prefix and force a mapping to an IP address that is embedded in the hostname. So for example:

    http-echo-service-http-ace-demo.23.72.153.69.nip.io
    will simply be routed to the 23.72.153.69 IP address.

    In order to get the values needed to configure this field if using nip.io you need to get the IP Address associated to the EKS cluster load balancer. You can use this command:

    Kubectl get services -n ingress-nginx -o wide
    This will result in something like the following output:

    NAME                                TYPE          CLUSTER-IP     EXTERNAL-IP                                    
    ingress-nginx-controller            LoadBalancer  10.100.88.168  abc123.elb.amazonaws.com   
    ingress-nginx-controller-admission  ClusterIP     10.100.143.60  <none>
    From the output grab the fully qualified domain name (FQDN) associated with the ingress-nginx-controller. This is the address which can be called from outside the cluster over the public internet.

    Run the nslookup command on the FQDN to get the IP address. For example, based on the preceding output, we would use the following command.

    nslookup abc123.elb.amazonaws.com
    The output might look something like this:

    Server: xxx.xxx.xxx.xxx
    Address: xxx.xxx.xxx.xxx::1111
    Non-authoritative answer:
    Name: abc123.elb.amazonaws.com
    Address: yyy.yyy.yyy.yyy
    From the output get the IP Address (yyy.yyy.yyy.yyy), and build up the nip.io URL like this:

    http-echo-service-http-ace-demo.yyy.yyy.yyy.yyy.nip.io
    This then is the value that you should use for <external-hostname> in the ingress definition file above.

  4. Deploy the ingress object using the following command:

    kubectl apply -f is-ingress.yaml
    Once the ingress configuration has been deployed your micro-integration is ready to receive traffic from outside the EKS cluster.

  5. Test the integration

    Verify the status of Integration Server pod and ensure that its status has changed to 1/1 Running.For example:

    $ kubectl get pods
    NAME                                       READY   STATUS    RESTARTS   AGE
    http-echo-service-is-64bc7f5887-g6dcd      1/1     Running   0          67s
    
    Invoke the service using curl command using the first URL you configured in the ingress definition. So if you have DNS set up:

    $ curl -X POST http://http-echo-service-http-ace-demo. yyy.yyy.yyy.yyy.yourdomain.com/Echo

    or if you don’t have DNS configured, then use the nip.io url we constructed earlier - replacing with the IP address you found for the:

    $ curl -X POST http://http-echo-service-http-ace-demo. yyy.yyy.yyy.yyy.nip.io/Echo

    You should receive a response similar to the following, letting you know that your request made it into the container and returned back.

    <echo>
          <datestamp>2021-10-30T06:03:59.717574Z</datestamp>
    </echo>

Conclusion

So that’s it, you’ve done it! You’ve deployed a (very) simple flow from an IBM Integration Bus environment into an IBM App Connect container on Amazon EKS. There are a few more steps compared to OpenShift, but most of those are up front to prepare the environment and are then in place for all subsequent deployments. There are one or two other differences worth noting when installing on a non-OpenShift based environment, and these are documented here.

The following diagram provides a summary of the activities and components we have deployed in the EKS cluster.


The steps would be largely the same for any non-OpenShift Kubernetes environment such as the Azure Red Hat OpenShift service, or indeed a self-managed non-OpenShift Kubernetes cluster.


Acknowledgement and thanks to Kim Clark for providing valuable input to this article.


#API Connect
#cloudintegration
#Automation
#AppConnect
#amazon

Permalink

Comments

Fri August 05, 2022 12:16 PM

Hello Joel,

About the App connect dashboard when deployed don Kubernetes (e.g. EKS / AKS), it is accessible freely & unsecured. So, any one knows the dashboard ingress URL can manipulate the deployed integration flows. 

How to deal with such cases ? 

In addition to that, how to secure the integration flows deployed as integration server resources using App connect operator ?

Thanks,
Sujeeth

Thu May 26, 2022 10:44 AM

Hi Sowmya,

Thank you for spotting the typo. I have updated the article to use the same namespace across the article.

I'll review where it makes sense to add the system requirements page and I'll update the article accordingly.

Best regards... Joel

Tue May 24, 2022 03:53 AM

Thank you for this Blog, Couple things to bring to your attention.
1. Can you stick the link to system requirements, as it helps?
2. Is that a typo? You meant ace-demo i guess
kubectl create namespace app-demo

Mon April 25, 2022 07:49 PM

Hi Purush, Thank you for your feedback. This post was focused on Toolkit based Integrations. We'll add the Designer Authoring Tool scenario to the list of future blogs, but in the meantime I can confirm  Designer is supported since it is one of the operands available as part of the App Connect Operator.

Best regards... Joel

Fri March 25, 2022 05:54 PM

Thanks for posting this Blog. I did not see steps for creating AppConnect Designer instance. Could you please confirm if AppConnect Designer instance supported on EKS?
Thanks