Introduction
This tutorial guides you through a step-by-step process of installing Apache Kafka on the Red Hat OpenShift Container platform on IBM Power. You can install any operator available in OperatorHub through cluster web console or command-line interface (CLI) to be able to use its functionality. The example in this tutorial explains how to install the serverless operator and the AMQ Streams operator through the OpenShift Container Platform.
What is Serverless?
Serverless is a Red Hat’s add-on feature that allows applications to be built and run without requiring an understanding of the infrastructure in which they are deployed. The serverless platform is simple to use and easily accessible to everyone. Developers can concentrate more on software development if they don't have to worry about infrastructure. For more details about serverless, refer Red Hat OpenShift Serverless on IBM Power - IBM Developer
What is Apache Kafka?
Apache Kafka is an event-streaming software platform that enables you to read, write, store, and process events (also called records or messages in the documentation) across many systems. Example of events is payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and so on. These events are organized and stored in topics. A topic is like a folder in a file system, and the events are the files in that folder.
It is based on a publish-subscribe (also referred to as pub/sub) messaging model and is designed to be fault-tolerant and scalable, and provide high throughput and low latency. Kafka started as a project on LinkedIn and was later open sourced to facilitate its adoption. It is written in Scala and Java, and it is a part of the open source Apache Software Foundation.
Serverless 1.22.0 started supporting Apache Kafka on the Red Hat OpenShift Container platform on Power. While AMQ is an independent operator since beginning, we have included Kafka support for serverless from this release onward.
The Red Hat AMQ streams component is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency.
AMQ Streams is based on Apache Kafka, a popular platform for streaming data delivery and processing, and makes it easy to run Apache Kafka on Red Hat OpenShift Container Platform.
AMQ Streams provides the following three operators:
- Cluster operator: Is responsible for deploying and managing Apache Kafka clusters within an OpenShift cluster.
- Topic operator: Is responsible for managing Kafka topics within a Kafka cluster running within an OpenShift cluster.
- User operator: Is responsible for managing Kafka users within a Kafka cluster running within an OpenShift cluster.
Prerequisites
A fully functional Red Hat OpenShift Container Platform must be available for Kafka installation.
Estimated time
If you have a functional Red Hat OpenShift Container Platform, then the following tasks can be completed in 2 hours approximately.
Steps
Complete the following steps to install Apache Kafka on Red Hat OpenShift Container Platform.
1. Install Red Hat Serverless operator on the OpenShift Container Platform
Kafka is installed in the knative-eventing namespace. Therefore, to create the knative-eventing namespace on OpenShift Container Platform, you have to install the RedHat Serverless operator. After the RedHat Serverless operator is successfully installed, it creates various resources (such as pods, deployments, replica sets) in the following namespaces:
- openshift-serverless,
- knative-serving and
- knative-eventing
2. Install AMQ Streams operator on OpenShift Container Platform
AMQ Streams is based on Apache Kafka and enables Apache Kafka to smoothly run on Red Hat OpenShift Container Platform. Perform the following steps to install the AMQ streams.
Note: Type all the commands that are provided in the following steps in the OpenShift Container Platform terminal.
- Install the general availability (GA) version of the AMQ Streams operator from the operator hub.
AMQSTREAMS_OPERATOR_SOURCE="${AMQSTREAMS_OPERATOR_SOURCE:-amqstreams-operator-operatorhub}"
- Run the following command to check whether the package manifest is available for the amq-streams (operator).
Information: The catalog source (broucher) contains package manifest which has or shows serverless-operator to be installed. The catalog source name should be matched with the contents in the package manifest.
Note: When you run the command, it provides the package manifest before 360 seconds. However, if the package manifest is not available within the given time, the installation process will be aborted.
wait_while 360 "[ \$(oc get packagemanifests -o name | grep -c '^packagemanifest.packages.operators.coreos.com/amq-streams') -lt 1 ]"
- Run the following command to confirm whether the status of the channel is stable. The command runs in less than 360 seconds (the maximum wait time is 360 seconds) to confirm the status. A stable status confirms the availability of the latest stable version of the operator.
wait_while 360 "! oc get packagemanifests amq-streams
-o=custom-columns=DEFAULT_CHANNEL:.status.defaultChannel
--no-headers=true | grep -qv '
- After the AMQ streams package manifest is confirmed, verify that the channel is made available by running following command.
$(oc get packagemanifests -o json | jq -r ".items[] |
select(.metadata.name == \"amq-streams\" and
.status.catalogSource != \"redhat-operators\") | .status.defaultChannel")
If the channel is not available, enable the default channel manually.
channel=${OPERATORS_CHANNEL:-stable}
- Run the following command to confirm if the catalog source is redhat operators because AMQ streams operator is a part of the redhat operators catalog.
if oc get packagemanifests amq-streams --no-headers=true -o=custom-columns=CATALOGSOURCE:.status.catalogSource >/dev/null; then
catalogSource=$(oc get packagemanifests amq-streams --no-headers=true -o=custom-columns=CATALOGSOURCE:.status.catalogSource)
catalog source is: redhat-operators
- Create a subscription for the AMQ streams operator as part of the operator installation process. The following YAML file creates a subscription for the AMQ streams operator in the kafka namespace as follows:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: amq-streams
namespace: kafka
spec:
channel: stable
installPlanApproval: Automatic
name: amq-streams
source: redhat-operators
sourceNamespace: openshift-marketplace
The following message confirms that the subscription is created successfully.
subscription.operators.coreos.com/serverless-operator created
- Run the following command to know the cluster service version (CSV) of the all the installed operators. The cluster service version (CSV) confirms the successful installation of the AMQ Streams operator.
$oc get csv
Note: The following screen capture displays the successful installation of the AMQ Streams operator.
- Ensure whether all the required resources are available in the respective namespaces and the AMQ Streams operator is installed successfully:
- Run the following command to validate the AMQ Streams CSV in the Kafka namespace.
$ oc get installplan -n kafka
NAME CSV APPROVAL APPROVED
install-smqn8 amqstreams.v2.1.0-6 Automatic true
- Run the following command to validate whether the subscription is available in the Kafka namespace.
$ oc get subscription amq-streams -n kafka
NAME PACKAGE SOURCE CHANNEL
amq-streams amq-streams redhat-operators stable
- Run the following command to specifically verify the AMQ Stream CSV.
$ oc get csv -n kafka
NAME DISPLAY VERSION REPLACES PHASE
amqstreams.v2.1.0-6 Red Hat Integration - AMQ Streams 2.1.0-6 amqstreams.v2.1.0-5 Succeeded
serverless-operator.v1.24.0 Red Hat OpenShift Serverless 1.24.0 serverless-operator.v1.23.0 Succeeded
- Pods are installed as a part of the AMQ Streams operator installation process. Run the following command to verify whether the pod was successfully installed in the Kafka namespace.
$ oc get pods -n kafka
NAME READY STATUS RESTARTS AGE
amq-streams-cluster-operator-v2.1.0-6-5f6cd8d47f-x6d8t 1/1 Running 0 34s
3. Install knative-kafka on Red Hat OpenShift Container Platform
After ensuring that the Red Hat OpenShift Serverless and AMQ Streams operators are installed successfully, you can now proceed to install the Apache Kafka project. The Apache Kafka project installed the knative-eventing namespace that was created after installing the OpenShift Serverless operator.
- Run the following command to ensure that the knative-eventing namespace is available. If for some unforeseen reason the knative-eventing namespace is not available, then you need to create it manually. For this instance, the knative-eventing namespace will already be available because you explicitly installed it during the OpenShift Serverless operator installation.
if! oc get namespace knative-eventing >/dev/null 2>&1; then
oc create namespace knative-eventing
- Run the following command in CLI. You need to wait until the command runs, to ensure that the KnativeKafka Custom Resource Definition (CRD) is available (the maximum wait time is 300 seconds). When the availability of KnativeKafka CRD is confirmed, it proceeds with the Apache Kafka project installation.
wait_while 300 "! oc get crd -o custom-columns=KIND:.status.acceptedNames.kind | grep KnativeKafka"
- Run the following YAML script to create knativeKafka resource pods in the knative-eventing namespace.
apiVersion: operator.serverless.openshift.io/v1alpha1
kind: KnativeKafka
metadata:
name: knative-kafka
namespace: knative-eventing
spec:
channel:
enabled: true
bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
source:
enabled: true
sink:
enabled: true
broker:
enabled: true
defaultConfig:
bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
- Run the following command in the CLI. Then, you need to wait until the following command runs to ensure that KnativeKafka is in the Ready The maximum wait time is 720 seconds.
$oc get knativekafkas.operator.serverless.openshift.io -n knative-eventing '-o=jsonpath={.items[0].status.conditions[?(@.type==\"Ready\")].status}' | grep 'True'"
- After validating that KnativeKafka is in the Ready state, run the following command to check whether the KnativeKafka resources pods are available in the knative-eventing namespace.
oc get pods -n knative-eventing
The following screen capture lists the pods that are installed in the process.
4. Confirm that Kafka is installed from the OpenShift Container Platform console
You need to log in to cluster web console in the administrator mode. Click Operators --> InstalledOperators. When the following screen is displayed, you can be assured that the Apache Kafka project is successfully installed.
Summary
This tutorial provided a detailed process explaining how to install the Apache Kafka project on IBM Power. As part of this process:
- You installed the OpenShift Serverless operator to get the knative-eventing namespace because the Apache Kafka project is in the knative-eventing namespace.
- Then you installed the AMQ Streams operator because it facilitates a higher throughput for Apache Kafka.
- Further you installed Knative Kafka and confirmed whether the installation was successful on the OpenShift Container Platform console.
You are welcome to drop in comments if you thought that the tutorial was informative. Also, let us know if you have any queries.
References