Event Streams: Apache Kafka for the Enterprise, now with Operators!
Operators for Kafka
What's so great about Operators?
Previously Event Streams was deployed using Helm charts which work well for installation and stateless applications, but doesn't cover ongoing operations or stateful applications well. Operators extend Kubernetes to be able to support stateful applications like Kafka, allowing you to use all the Kubernetes tooling that you're used to but fully integrated with Kafka concepts.
The term 'Operator' is a neat use of language: Operators allow the system to behave like a human operator who has deep knowledge of how the system ought to behave.
The Operator understands Kafka concepts like users, partitions and topics. That means if you ask it (by declaring it in a yaml file) to perform an update then it will use its knowledge of the Kafka cluster to perform the update in a controlled way, thinking about all the different pieces like Zookeeper, brokers and so on.
Operators go beyond installation to a truly declarative deploy - you describe how you want the system to look and behave in a Custom Resource (CR). Perhaps you want to increase the number of partitions for your Kafka deployment. You simply change the CR to say how many partitions you want - the Operator is watching to see for any new or changed CRs and then responds to make change the deployment to fit. This kind of behaviour is often called day-2 operations, taking you beyond the day-1 of installation.
Infrastructure as code
Operators marks a real step towards infrastructure-as-code. You can now put the yaml files (CRs) that define how to build your infrastructure into source code control alongside your application, including everything that the application needs to run like users and topics. Everything that's needed to build or rebuild a fully working application stack is right there - so you could issue a single kubectl apply and deploy everything!
Built on Open Source
The Event Streams Operator is built on the Open Source Strimzi Operators for Kafka project at http://strimzi.io. Strimzi is an incubator project at the Cloud Native Computing Foundation. Event Streams extends Strimzi to include additional features like UI and Schema registry.
Getting started with IBM Event Streams
So you'd like to try out IBM Event Streams?
There are a few great places that you can go to get started.
Previously we offered an unsupported Community Edition of Event Streams, based on IBM Cloud Private. As we move to more native deployment of Event Streams on OpenShift with Operators, we are moving to the above options as ways to get started with Kafka and Event Streams.
*1. At time of writing the tutorials are still based upon Event Streams on IBM Cloud Private. These will be updated following the GA of Cloud Pak for Integration 2020.2.1