Containers, Kubernetes, OpenShift on Power

 View Only

Apache Kafka on IBM Power

By Satish Waghole posted Thu January 12, 2023 12:53 AM

  

Introduction

This tutorial guides you through a step-by-step process of installing Apache Kafka on the Red Hat OpenShift Container platform on IBM Power. You can install any operator available in OperatorHub through cluster web console or command-line interface (CLI) to be able to use its functionality. The example in this tutorial explains how to install the serverless operator and the AMQ Streams operator through the OpenShift Container Platform.

What is Serverless?

Serverless is a Red Hat’s add-on feature that allows applications to be built and run without requiring an understanding of the infrastructure in which they are deployed. The serverless platform is simple to use and easily accessible to everyone. Developers can concentrate more on software development if they don't have to worry about infrastructure. For more details about serverless, refer Red Hat OpenShift Serverless on IBM Power - IBM Developer

What is Apache Kafka?

Apache Kafka is an event-streaming software platform that enables you to read, write, store, and process events (also called records or messages in the documentation) across many systems. Example of events is payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and so on. These events are organized and stored in topics. A topic is like a folder in a file system, and the events are the files in that folder.

It is based on a publish-subscribe (also referred to as pub/sub) messaging model and is designed to be fault-tolerant and scalable, and provide high throughput and low latency. Kafka started as a project on LinkedIn and was later open sourced to facilitate its adoption. It is written in Scala and Java, and it is a part of the open source Apache Software Foundation.

Serverless 1.22.0 started supporting Apache Kafka on the Red Hat OpenShift Container platform on Power. While AMQ is an independent operator since beginning, we have included Kafka support for serverless from this release onward.

The Red Hat AMQ streams component is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency. 

AMQ Streams is based on Apache Kafka, a popular platform for streaming data delivery and processing, and makes it easy to run Apache Kafka on Red Hat OpenShift Container Platform.

AMQ Streams provides the following three operators:

  • Cluster operator: Is responsible for deploying and managing Apache Kafka clusters within an OpenShift cluster.
  • Topic operator: Is responsible for managing Kafka topics within a Kafka cluster running within an OpenShift cluster.
  • User operator: Is responsible for managing Kafka users within a Kafka cluster running within an OpenShift cluster.

Prerequisites

A fully functional Red Hat OpenShift Container Platform must be available for Kafka installation.

Estimated time

If you have a functional Red Hat OpenShift Container Platform, then the following tasks can be completed in 2 hours approximately.

Steps

Complete the following steps to install Apache Kafka on Red Hat OpenShift Container Platform.

1. Install Red Hat Serverless operator on the OpenShift Container Platform

Kafka is installed in the knative-eventing namespace. Therefore, to create the knative-eventing namespace on OpenShift Container Platform, you have to install the RedHat Serverless operator. After the RedHat Serverless operator is successfully installed, it creates various resources (such as pods, deployments, replica sets) in the following namespaces:

  • openshift-serverless,
  • knative-serving and
  • knative-eventing

2. Install AMQ Streams operator on OpenShift Container Platform

    AMQ Streams is based on Apache Kafka and enables Apache Kafka to smoothly run on Red Hat OpenShift Container Platform. Perform the following steps to install the AMQ streams.

    Note: Type all the commands that are provided in the following steps in the OpenShift Container Platform terminal.

    1. Install the general availability (GA) version of the AMQ Streams operator from the operator hub.
      AMQSTREAMS_OPERATOR_SOURCE="${AMQSTREAMS_OPERATOR_SOURCE:-amqstreams-operator-operatorhub}"
    2. Run the following command to check whether the package manifest is available for the amq-streams (operator).

      Information: The catalog source (broucher) contains package manifest which has or shows serverless-operator to be installed. The catalog source name should be matched with the contents in the package manifest.

      Note: When you run the command, it provides the package manifest before 360 seconds. However, if the package manifest is not available within the given time, the installation process will be aborted.

      wait_while 360 "[ \$(oc get packagemanifests -o name | grep -c '^packagemanifest.packages.operators.coreos.com/amq-streams') -lt 1 ]"
    3. Run the following command to confirm whether the status of the channel is stable. The command runs in less than 360 seconds (the maximum wait time is 360 seconds) to confirm the status. A stable status confirms the availability of the latest stable version of the operator.
      wait_while 360 "! oc get packagemanifests amq-streams
      -o=custom-columns=DEFAULT_CHANNEL:.status.defaultChannel 
      --no-headers=true | grep -qv '
    4. After the AMQ streams package manifest is confirmed, verify that the channel is made available by running following command.
      $(oc get packagemanifests -o json | jq -r ".items[] |
          select(.metadata.name == \"amq-streams\" and 
      .status.catalogSource != \"redhat-operators\") |  .status.defaultChannel")
      

      If the channel is not available, enable the default channel manually.

      channel=${OPERATORS_CHANNEL:-stable}
      
    5. Run the following command to confirm if the catalog source is redhat operators because AMQ streams operator is a part of the redhat operators catalog.
      if oc get packagemanifests amq-streams --no-headers=true -o=custom-columns=CATALOGSOURCE:.status.catalogSource >/dev/null; then
      catalogSource=$(oc get packagemanifests amq-streams --no-headers=true -o=custom-columns=CATALOGSOURCE:.status.catalogSource)
      
      catalog source is: redhat-operators
      
    6. Create a subscription for the AMQ streams operator as part of the operator installation process. The following YAML file creates a subscription for the AMQ streams operator in the kafka namespace as follows:
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
          name: amq-streams
          namespace: kafka
      spec:
          channel: stable
                  installPlanApproval: Automatic
          name: amq-streams
                  source: redhat-operators
          sourceNamespace: openshift-marketplace       
      

      The following message confirms that the subscription is created successfully.

      subscription.operators.coreos.com/serverless-operator created
    7. Run the following command to know the cluster service version (CSV) of the all the installed operators. The cluster service version (CSV) confirms the successful installation of the AMQ Streams operator.
      $oc get csv

      Note: The following screen capture displays the successful installation of the AMQ Streams operator.

      successful installation of the AMQ Streams operator
    8. Ensure whether all the required resources are available in the respective namespaces and the AMQ Streams operator is installed successfully:
      • Run the following command to validate the AMQ Streams CSV in the Kafka namespace.
        $ oc get installplan -n kafka
        NAME            CSV                   APPROVAL    APPROVED
        install-smqn8   amqstreams.v2.1.0-6   Automatic   true
        
      • Run the following command to validate whether the subscription is available in the Kafka namespace.
        $ oc get subscription amq-streams -n kafka
        NAME          PACKAGE       SOURCE             CHANNEL
        amq-streams   amq-streams   redhat-operators   stable
        
      • Run the following command to specifically verify the AMQ Stream CSV.
        $ oc get csv -n kafka
        NAME                          DISPLAY                             VERSION   REPLACES                      PHASE
        amqstreams.v2.1.0-6           Red Hat Integration - AMQ Streams   2.1.0-6   amqstreams.v2.1.0-5           Succeeded
        serverless-operator.v1.24.0   Red Hat OpenShift Serverless        1.24.0    serverless-operator.v1.23.0   Succeeded
        
      • Pods are installed as a part of the AMQ Streams operator installation process. Run the following command to verify whether the pod was successfully installed in the Kafka namespace.
        $ oc get pods -n kafka
        NAME                                                     READY   STATUS    RESTARTS   AGE
        amq-streams-cluster-operator-v2.1.0-6-5f6cd8d47f-x6d8t   1/1     Running   0          34s
        

    3. Install knative-kafka on Red Hat OpenShift Container Platform

    After ensuring that the Red Hat OpenShift Serverless and AMQ Streams operators are installed successfully, you can now proceed to install the Apache Kafka project. The Apache Kafka project installed the knative-eventing namespace that was created after installing the OpenShift Serverless operator.

    1. Run the following command to ensure that the knative-eventing namespace is available. If for some unforeseen reason the knative-eventing namespace is not available, then you need to create it manually. For this instance, the knative-eventing namespace will already be available because you explicitly installed it during the OpenShift Serverless operator installation.
      if! oc get namespace knative-eventing >/dev/null 2>&1; then
      oc create namespace knative-eventing
      
    2. Run the following command in CLI. You need to wait until the command runs, to ensure that the KnativeKafka Custom Resource Definition (CRD) is available (the maximum wait time is 300 seconds). When the availability of KnativeKafka CRD is confirmed, it proceeds with the Apache Kafka project installation.
      wait_while 300 "! oc get crd -o custom-columns=KIND:.status.acceptedNames.kind | grep KnativeKafka"       
    3. Run the following YAML script to create knativeKafka resource pods in the knative-eventing namespace.
      apiVersion: operator.serverless.openshift.io/v1alpha1
      kind: KnativeKafka
      metadata:
      name: knative-kafka
      namespace: knative-eventing
      spec:
          channel:
              enabled: true
              bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
          source:
              enabled: true
          sink:
              enabled: true
          broker:
              enabled: true
              defaultConfig:
                  bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092                  
    4. Run the following command in the CLI. Then, you need to wait until the following command runs to ensure that KnativeKafka is in the Ready The maximum wait time is 720 seconds.
      $oc get knativekafkas.operator.serverless.openshift.io -n knative-eventing '-o=jsonpath={.items[0].status.conditions[?(@.type==\"Ready\")].status}' | grep 'True'"         
    5. After validating that KnativeKafka is in the Ready state, run the following command to check whether the KnativeKafka resources pods are available in the knative-eventing namespace.
      oc get pods -n knative-eventing            

      The following screen capture lists the pods that are installed in the process.

    4. Confirm that Kafka is installed from the OpenShift Container Platform console

    You need to log in to cluster web console in the administrator mode. Click Operators --> InstalledOperators. When the following screen is displayed, you can be assured that the Apache Kafka project is successfully installed.

    step 4

    Summary

    This tutorial provided a detailed process explaining how to install the Apache Kafka project on IBM Power. As part of this process:

    1. You installed the OpenShift Serverless operator to get the knative-eventing namespace because the Apache Kafka project is in the knative-eventing namespace.
    2. Then you installed the AMQ Streams operator because it facilitates a higher throughput for Apache Kafka.
    3. Further you installed Knative Kafka and confirmed whether the installation was successful on the OpenShift Container Platform console.

    You are welcome to drop in comments if you thought that the tutorial was informative. Also, let us know if you have any queries.

    References

    Permalink