Apache Kafka is a widely-used distributed streaming platform that has been used in many different use cases to provide a reliable, scalable, and high-performance messaging system. One of the core components of Kafka has been ZooKeeper,which is used to manage configuration information, provide distributed synchronization, and detect failures. With the release of Kafka Improvement Proposal KIP-500, ZooKeeper can be replaced with a self-managed metadata quorum Kafka Raft (Kraft), which provides benefits over the old system.
In this blog, we will explore where this matters and if you should consider moving away from ZooKeeper. As the number of Kafka brokers and topics increases, the dependence on ZooKeeper can create bottlenecks that can impact Kafka's performance and scalability.
This feature is particularly useful in scenarios where Kafka is deployed at a large scale. It replaces ZooKeeper with a self-managed metadata quorum, which is a set of Kafka brokers that manages metadata. The metadata quorum is responsible for managing configuration information, topic and partition information, and offset information. The metadata quorum uses a consensus protocol called Kafka Raft (KRaft) to ensure that all brokers have the same view of metadata. In addition, KIP-500 provides a new administrative API for managing metadata, which simplifies the management of Kafka clusters.
Lets look at the benefits if you move away from ZooKeeper to ZooKeeper-less KRaft based Kafka deployment -
Simplifies the deployment: By removing ZooKeeper, Kafka becomes a simpler system with fewer moving parts. This can make it easier to manage and reduce the potential for misconfiguration and errors.
Improves scalability and performance: With ZooKeeper removed, Kafka can now scale to support many more topics and partitions than before. This is because the ZooKeeper metadata limits that were previously in place are no longer a factor. It leads to better resource utilization and improved performance.
More reliable: With ZooKeeper, Kafka had a single point of failure, which could lead to downtime and data loss. This feature provides a more reliable metadata management system by replicating metadata across the metadata quorum.
Reduction in shutdown and recover time: It provides a fast metadata recovery mechanism that can recover metadata within a few seconds, which is a significant improvement over ZooKeeper's recovery time of several minutes.
Reduced resources: Eliminating the need for a ZooKeeper ensemble means less memory and fewer CPUs are required to run your Kafka cluster. As ZooKeeper requires storage, it also means fewer persistent volumes are required. All of this can bring cost savings for running the cluster.
If the issues mentioned align with your challenges, it may be worthwhile to and begin streamlining your existing Kafka deployments. Thorough planning and careful consideration of your business requirements and priorities are necessary, particularly since the benefits of moving away from ZooKeeper become more apparent for larger and newer Kafka deployments .
IBM Event Streams latest release 11.1.5 introduces KRaft based deployment (zookeeper less) for new installation under Technology Preview.As of now, it is intended for testing purposes only and not for production use. You can learn more about it in the IBM Event Streams documentation. Do give it a try and leave any questions you may have in the comments section.
------------------------------
Adeoye Omoboya
------------------------------