App Connect Enterprise (ACE) solutions can be built and deployed via standard pipelines such as Jenkins and Tekton with runtime targets including containers and software-as-a-service (SaaS) as well as integration nodes, but there is a common misconception that the container cluster has to be a large multi-node installation for this to be successful. The reality is more encouraging for experimenters and other ACE developers: these pipelines can in fact be run locally on small VMs without requiring enterprise-grade infrastructure.
Despite the apparent need for a large OpenShift or other Kubernetes cluster to run Tekton, it turns out that Tekton pipelines and ACE runtime can be hosted in a local VM with as little as 8GB of RAM using Minikube as the container platform. With a bit more RAM, the ACE operator used in the Cloud Pak for Integration (CP4i) can also be run locally and used to provision IntegrationRuntime resources, providing greater flexibility in local testing of configurations and deployment YAML. The Dashboard can be used locally, with all the basic functionality being available to use.
Minikube is not intended to be a production platform (they say in the docs they "proudly focus on helping application developers and new Kubernetes users"), but does allow individual developers the chance to experiment in ways that would be difficult with clusters shared among many people, and this blog post describes how to do this using the developer edition of ACE (no purchase necessary!) as well as using certified containers and the ACE operator.
Expected use
Minkube-based pipelines are best thought of as a local alternative to an official deploy pipeline, using the same technology (Tekton, ACE, etc) but not requiring the same resources:
This local pipeline might be used as a prototype for the official pipeline (so the Minikube pipeline would be created first) or else as a safe way to try out modifications to an official pipeline, with the Minikube pipeline replicating the existing official pipeline. Experiments with serverless ACE are also achievable, with Knative serving being available in Minikube.
While this approach works well for debugging Tekton YAML, deployment scripts, serverless, and other ACE-related configuration, there are clearly limitations on how far this can go: OpenShift Routes do not exist in Minikube (more on this later), security configuration will be different, and so on. However, the ability to run full pipelines and create deployment scripts in isolation is still likely to be helpful, and this can all be run on much smaller systems than RedHat single-node OpenShift (which requires a minimum 8 CPU cores and 32GB RAM).
This blog post uses the ACE demo pipeline for convenience, though the technology is not limited to that example. The descriptions of setup, pipelines, etc are deliberately abbreviated here to save space, and the main instructions for the various components are at https://github.com/ot4i/ace-demo-pipeline plus the CP4i ACE Operator setup instructions in the main product docs.
Initial setup
Minikube has setup instructions for various operating systems, but this example will be using Ubuntu 22.04 so apt install minikube on a Docker-enabled Ubuntu VM is all we need to do to get going. The VM in this case has 8GB of RAM, which is enough for the Docker driver (and less would work for some use cases) but did not appear to work for the qemu2 driver, so it's worth making sure Docker is mentioned in the initial startup:
The examples shown is following the instructions at https://github.com/ot4i/ace-demo-pipeline/blob/main/tekton/minikube/README.md and specifies the --insecure-registry parameter to make image pushes simpler during pipeline runs. Note the mention of docker on the second line, which indicates Docker is installed and accessible: the user running Minikube must be able to access /var/run/docker.sock (default location) for this to succeed.
The ACE demo pipeline instructions suggest installing the dashboard, registry, and metrics-server addons, and eventually also suggest installing ingress. However, while it is possible to use Kubernetes ingress to access the Kubernetes dashboard and ACE applications, it is often easier to simply use Kubernetes port forwarding to the VM itself: running
kubectl -n kubernetes-dashboard port-forward --address 0.0.0.0 svc/kubernetes-dashboard 9090:80
allows access to the Kubernetes dashboard using port 9090 on the VM address (such as http://192.168.1.240:9090) without needing to configure any ingress solution, and this approach works for other services also.
Running the Tekton pipeline
The ACE demo pipeline includes instructions for configuring and running on Minikube, including installing Tekton and configuring the pipeline. The choice of container base image is important (see blog post here for more detail), with the ace container being the obvious choice if an IBM entitlement key is available (paying customers should have one) and ace-minimal if not; the latter is built from the Developer edition of ACE that can be downloaded free of charge.
Once the Tekton dashboard has been installed (as part of the instructions), then running port forwarding is the simplest way to gain visibility into Tekton pipelines and runs; similar to the Kubernetes dashboard above, running
kubectl --namespace tekton-pipelines port-forward --address 0.0.0.0 svc/tekton-dashboard 9097:9097
makes the Tekton dashboard accessible on the VM network address:
Once the pipelines have completed and the solution is deployed, it should be accessible via port forwarding or ingress, and Knative serverless can also be enabled (see the Knative setup section in the demo pipeline Minikube README) after Kourier is installed and the pipeline can then be re-run with knativeDeploy set to true. Even after this, there should be plenty of headroom on the Minikube VM, as the memory consumption is likely to be around 2GB for everything (including Knative):
CP4i
As well as using the ACE product by itself, it is also possible to install the ACE operator used as part of CP4i. The installation instructions follow the "Kubernetes" section of the installation docs at https://www.ibm.com/docs/en/app-connect/containers_cd?topic=connect-installing-uninstalling-kubernetes with the usual requirement for an IBM Entitlement Key and so on. The installation required the Minikube VM to have 16GB of RAM available to it (8GB led to swapping during operator SDK installation for reasons that are unclear) but other than that it worked very much as shown in the docs.
The approximate list of commands is as follows (the docs will contain the latest instructions) using files from the ACE demo pipeline repo and following https://github.com/ot4i/ace-demo-pipeline/blob/main/tekton/os/cp4i/minikube/README.md:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
kubectl get pods --namespace cert-manager
kubectl patch deployment \
cert-manager \
--namespace cert-manager \
--type='json' \
-p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
"--v=2",
"--cluster-resource-namespace=$(POD_NAMESPACE)",
"--leader-election-namespace=kube-system",
"--enable-certificate-owner-ref"
]}]'
operator-sdk olm install
kubectl create namespace cp4i
kubectl apply -f tekton/os/cp4i/minikube/minikube-install-og.yaml
kubectl apply -f tekton/os/cp4i/minikube/minikube-install-catalog-source.yaml
kubectl get CatalogSources ibm-appconnect-catalog -n olm
kubectl create secret -n cp4i docker-registry ibm-entitlement-key --docker-username=cp --docker-password=IBMENTITLEMENTKEY --docker-server=cp.icr.io
kubectl apply -f tekton/os/cp4i/minikube/minikube-install-subscription.yaml
and once this has completed then it should be possible to create a Dashboard using the example YAML:
kubectl apply -f tekton/os/cp4i/minikube/minikube-install-dashboard.yaml
which will create a dashboard without using the IBM CommonServices layer (which has not been configured), and at that point port forwarding can be used to access the ACE dashboard or tekton/os/cp4i/minikube/minikube-dashboard-ingress.yaml can be used to configure ingress.
Issues creating ACE runtimes
Attempting to create an ACE IntegrationRuntime from the dashboard (or from a pipeline) is likely to fail, however, as the default at 12.0.12 is to instruct the operator to configure Prometheus metrics gathering for the ACE server, and Prometheus is not installed. This leads to ACE pods not being created because the IntegrationRuntime is stuck in "Pending" state:
$ kubectl describe integrationruntimes.appconnect.ibm.com tea-tekton-cp4i
Name: tea-tekton-cp4i
Namespace: cp4i
...
Status:
Available Replicas: 0
Conditions:
Last Transition Time: 2024-05-16T21:43:56Z
Message: Creating Service Metrics.
Reason: CreatingServiceMetrics
Status: True
Type: Pending
Custom Images: true
Phase: Pending
...
and the operator logs will show the problem (scroll to the right for the error on the last line:
$ kubectl logs ibm-appconnect-operator-754c4d9c4c-7pcfg
ts=2024-05-16T21:47:44.422225695Z level=info logger=controller.integrationruntime msg="Reconciling IntegrationRuntime" Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
ts=2024-05-16T21:47:44.422329792Z level=info logger=controller.integrationruntime msg="12.0.12.0-r2 reconcile" Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
ts=2024-05-16T21:47:44.443789267Z level=info logger=controller.integrationruntime msg="Successfully reconciled Service Account." Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
ts=2024-05-16T21:47:44.444106858Z level=info logger=controller.integrationruntime msg="Successfully reconciled Role." Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
ts=2024-05-16T21:47:44.444462248Z level=info logger=controller.integrationruntime msg="Successfully reconciled Role Binding." Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
ts=2024-05-16T21:47:44.452950301Z level=info logger=controller.integrationruntime msg="Successfully reconciled Network Policy." Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
ts=2024-05-16T21:47:44.469107331Z level=info logger=controller.integrationruntime msg="Failed to create or update Service Monitor named [tea-tekton-cp4i-metrics-monitor]. Retrying reconciliation: the server could not find the requested resource (post servicemonitors.monitoring.coreos.com)" Request.Namespace=cp4i Request.Name=tea-tekton-cp4i
The most permanent solution is to install Prometheus (see https://github.com/prometheus-operator/kube-prometheus#quickstart):
git clone https://github.com/prometheus-operator/kube-prometheus
cd kube-prometheus
kubectl apply --server-side -f manifests/setup
and another is to switch off metrics in the IntegrationRuntime YAML (which needs to be done every time):
metrics:
disabled: true
and then the ACE applications should be running and visible in the dashboard:
While the application is running successfully and the pods are up as expected, the underlying IntegrationRuntime custom resource (CR) will still show a status of "Warning" rather than the usual "Ready" status, and this is because the operator was unable to create OpenShift Routes for the application:
ubuntu@minikube-20231123:~/github.com/ace-demo-pipeline$ kubectl get -n cp4i IntegrationRuntime/ir-01-quickstart
NAME RESOLVEDVERSION STATUS REPLICAS AVAILABLEREPLICAS URL AGE CUSTOMIMAGES
ir-01-quickstart 12.0.12.0-r2 Warning 1 1 http:// 12m false
ubuntu@minikube-20231123:~/github.com/ace-demo-pipeline$ kubectl get -n cp4i IntegrationRuntime/ir-01-quickstart -o yaml | yq .status.conditions
- lastTransitionTime: "2024-05-29T14:17:39Z"
message: IntegrationRuntime is ready.
reason: Deployed
status: "True"
type: Ready
- lastTransitionTime: "2024-05-29T14:11:57Z"
message: No routes were created because routes are not available on this platform. You must configure external access manually.
reason: NoRoutesCreated
status: "True"
type: Warning
External connectivity requires either ingress configuration or port forwarding, with both allowing for access to the application.
Further configuration is also possible, including all the standard CP4i configurations (setdbparms.txt, server.conf.yaml, etc) and the full set of ACE capabilities. Note that while Minikube meets the "Minimum requirements" described at https://www.ibm.com/docs/en/app-connect/containers_cd?topic=kubernetes-supported-operating-environment it is not expected to be used as a production environment, and another environment (such as RedHat OpenShift) should be used for production deployments.
Conclusion
Despite not being a common production runtime, Minikube can run ACE servers, Tekton pipelines, Knative serverless, and much more without requiring significant investment in container cluster hardware. The ACE operator used in CP4i can also be brought up in slightly larger VMs, and overall Minikube provides a good way to experiment with ACE without needing any product licenses or cloud accounts.
#Highlights-home