The OpenMessaging Benchmark Framework is typically used to benchmark messaging systems in the cloud, but in this post I want to show how useful it can also be for Kafka clusters that you run yourself using IBM Event Streams.
From openmessaging.cloud:
The OpenMessaging Benchmark Framework is a suite of tools that make it easy to benchmark distributed messaging systems.
As I’ve written about before (when illustrating the impact of setting quotas at the Kafka cluster level, and when adding quotas at the event gateway level), Apache Kafka comes with a good performance test tool. That is still my go-to option if I just want an easy way to push data through a Kafka cluster in bulk.
But – OpenMessaging’s benchmark has some interesting features that make it a useful complement and worth considering.
The benefit that OpenMessaging talk about the most is that can be used with a variety of messaging systems, such as RocketMQ, Pulsar, RabbitMQ, NATS, Redis and more – although in this post I’m only interested in using it to benchmark an Apache Kafka cluster.
More interesting for me was their focus on realistic workloads rather than relying on static data.
Quoting again from openmessaging.cloud:
Benchmarks should be largely oriented toward standard use cases rather than bizarre edge cases
I also like the way that the framework orchestrates running distributed benchmarks. It deploys a “driver” pod and multiple “worker” pods into your Kubernetes cluster.
You open a terminal into the driver pod, and start a benchmark by running a command-line command. The driver pod makes HTTP calls to each of worker pods to get them all to participate in a distributed benchmark workload, and it collects the results from each worker into a single collated file for you.
It also starts each benchmark by getting all of the worker pods to participate in a warmup phase, before it starts the actual measured benchmark.
This all works well and it is a very simple system.
Specifying a benchmark
The command you run is just:
benchmark \
--workers $WORKERS \
--drivers DRIVER-NAME-HERE \
WORKLOAD-NAME-HERE
The workers flag is where you provide the location of your worker pods, but the driver pod has the environment variable WORKERS automatically configured for you, so that bit is easy.
The only decisions you need to make are what driver and workload to use for your benchmark. Each one is defined as a simple, human-readable YAML file containing standard Kafka client configuration properties so you can add your own if you want.
Choosing a driver
The driver option is where you specify how you want the Kafka clients used in your benchmark to behave.
For example, do you want the clients optimised for high throughput (kafka-throughput.yaml) or low latency (kafka-latency.yaml)?
Do you want clients to use a compression algorithm (e.g. kafka-compression-gzip.yaml, kafka-compression-zstd.yaml, kafka-compression-lz4.yaml)?
Do you want the clients to wait for and produce large batches (e.g. kafka-big-batches.yaml) or send events immediately without waiting (kafka-no-linger.yaml)?
Do you want the clients to optimise for preventing message duplication due to network retries (kafka-exactly-once.yaml)?
And so on – you can see the drivers available for Kafka benchmarks in the driver-kafka folder.
Choosing a workload
The workload option is where you specify what you want the Kafka clients to do.
For example, how many topics do you want it to use? How many partitions should the topics have? How many consumers and producers do you want to run? Do you want the producers to run at a fixed rate, or to produce as fast as possible? What size messages do you want the producers to create? Do you want to produce fixed static data, or to generate random data? How long do you want the benchmark to run for? There are many choices!
Each workload is defined as a human-readable YAML file – try clicking on one of the example workloads in the scrollable list below to see what they look like:
This means that running a benchmark looks something like:
benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-exactly-once.yaml \
workloads/1-topic-1-partition-1kb.yaml
The permutations enabled by the choice of drivers (how the Kafka clients should behave) and workloads (what the Kafka clients should do) makes for a large range of possible benchmarks you can run.
(And as all of these are defined in simple YAML files, it’s simple for you to define additional custom benchmarks.)
To illustrate the huge difference that your choice makes, I picked a half-dozen example combinations and ran them against my simple three-broker Kafka cluster.
I’ve only picked out a couple of metrics for each of my benchmarks (messasges per second and MB per second) but the benchmark framework collects much much more than this.
Click on the “results” link in each row to see the full detail
How to use a benchmark
The point I’m highlighting here is that there isn’t a single benchmark that gives you a definitive single result for how fast your Kafka cluster can go.
Different benchmarks will give you an indication of how your Kafka cluster could perform to support different types of workloads. By choosing (or writing) an appropriate driver and workload, you can get a tailored benchmark that reflects your own use case.
This can be useful in an absolute way (validating how your Kafka cluster will support your expected workload). It can be particularly useful in a relative way, by comparing the results from running the same benchmark in two different environments (such as comparing a dev/test environment with a prod environment, or comparing an old environment with a new environment ahead of a migration).
Running a benchmark
Finally, I’ll end with a more detailed walkthrough for how to run a benchmark – by showing you how I got the results I linked to above.
step 0 – you need a Kafka cluster
I did this both using Strimzi and IBM Event Streams. If you want to repeat what I’ve done below, you can use my Strimzi specification, or my Event Streams specification.
step 1 – clone the benchmark repository
git clone https://github.com/openmessaging/benchmark.git
cd benchmark
step 2 – update as needed
I made two tweaks:
I updated the version of Java from Java 8 to Java 17 (commit).
I updated the bootstrap.servers address in all of the driver-kafka files to point at my Kafka cluster (commit). If you need to add any additional config to connect to your Kafka cluster (e.g. configure authentication with sasl.jaas.config) then you should do that as well.
step 3 – build the benchmark applications
I used Java 17 for this, too.
export JAVA_HOME=$(/usr/libexec/java_home -v 17)
export PATH=$JAVA_HOME/bin:$PATH
mvn install -DskipTests
step 4 – build a container image with the benchmark applications
You need an image registry to host the image. This could be a public registry like quay or dockerhub, but I hosted my container image on the OpenShift cluster where I ran the benchmark.
oc login ...
oc new-project openmessaging
export REGISTRY=`oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}'`
docker login -u `oc whoami` -p `oc whoami --show-token` ${REGISTRY}
Wherever you put your image, set the REGISTRY environment variable to match for the commands below.
export BENCHMARK_TARBALL=package/target/openmessaging-benchmark-0.0.1-SNAPSHOT-bin.tar.gz
docker build --platform linux/amd64 --build-arg BENCHMARK_TARBALL . -f docker/Dockerfile -t ${REGISTRY}/openmessaging/benchmark:latest
docker push ${REGISTRY}/openmessaging/benchmark:latest
step 5 – deploy the benchmark pods
You will need to update the benchmark Helm chart to point at the image you built in step 4. (commit)
Once you’ve done this, it’s just a helm install command to get the benchmark pods ready.
cd deployment/kubernetes/helm
helm install --namespace openmessaging benchmark benchmark
step 6 – run your benchmark
You can exec into the driver pod to start your benchmark.
oc exec -n openmessaging -it benchmark-driver -- bash
For example, I ran:
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-throughput.yaml \
workloads/1-topic-16-partitions-1kb.yaml
and copied the results to a gist
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-compression-zstd.yaml \
workloads/max-rate-20-topics-1-partition-100b.yaml
and copied the results to a gist
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-big-batches-gzip.yaml \
workloads/max-rate-1-topic-10-partitions-10p-10c-1kb.yaml
and copied the results to a gist
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-big-batches.yaml \
workloads/1-topic-16-partition-100b.yaml
and copied the results to a gist
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-throughput.yaml \
workloads/max-rate-1-topic-10-partitions-1p-1c-100b.yaml
and copied the results to a gist
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-exactly-once.yaml \
workloads/1-topic-1-partition-1kb.yaml
and copied the results to a gist
bin/benchmark \
--workers $WORKERS \
--drivers driver-kafka/kafka-compression-lz4.yaml \
workloads/max-rate-10-topics-1-partition-64kb.yaml
and copied the results to a gist
You get the idea… Now try it for yourself!