Complex Kafka Troubleshooting Tasks Made Easy

One of the main objectives of CP4AIOps MustGather (download link) is to simplify complex troubleshooting tasks through automation.

Kafka, being the "heart and soul" of CP4AIOps, is known to be a "tough nut to crack" when it comes to troubleshooting it.

Tasks like checking for message count of all topics, extract the latest X number of messages from all/certain topics, or listing file size of log segments may look easy but they are not. 

To start, you would need to know in which namespace CP4AIOps is installed and which product version is in used.
After that, you would need to determine the broker details and then obtain username, password and certificate from secret, etc. to get connected to Kafka.
With CP4AIOps MustGather, all the above are handled for you automatically. It is now "worry-free", because there is no longer a need to keep multiple copies of Kafka connection script for different CP4AIOps versions or having to ask customer/peer on which namespace CP4AIOps is installed, etc. **

Now, let's explore several Kafka-related data collection tasks...
NOTE(i): All examples were executed on CP4AIOps 4.6.0.
NOTE(ii): "kafkacat" or "kcat" (link) must be installed locally.

(1) To display message count for all Kafka topics

(a) If you were in a remote session and need to check the message count of all Kafka topics on an ad-hoc basis:

waiops-mustgather.sh -V kafka-topics

(b) If you would like to have the message count for all Kafka topics to be included with other CP4AIOps MustGather data collection, you can achieve that with the "aimanager" plugin:

waiops-mustgather.sh -k cm,secret -P aimanager -Z 'COLLECT_ONLY=kafka'

NOTE(i): The command above collects data for configmap, secret and also message count of all Kafka topics.
NOTE(ii): You can find the details of message count in 0-README-FIRST/SUMMARY.log of the output file.

(2) To extract raw messages from Kafka topics

If you managed to obtain approval from customer to extract raw messages from Kafka topics, you can make use of the "aimanager" plugin to dump those data out.

NOTE: With all the following commands, the user will be prompted to agree to Kafka data extraction. To skip the prompt, add '-Y' (auto-yes) to the command.

(a) Latest 50 raw messages (default)

waiops-mustgather.sh -P aimanager -Z 'COLLECT_ONLY=kafka##DATA_DUMP=1

(b) Raw messages of the previous day

waiops-mustgather.sh -P aimanager -Z 'COLLECT_ONLY=kafka##DATA_DUMP=1##AIMGR_KAFKA_SEARCH_TOPIC_DIRECTION=latest'

NOTE: Supported options for AIMGR_KAFKA_SEARCH_TOPIC_DIRECTION are "beginning", "end", "latest (latest day)" and "hour (latest hour)" (default = end).

(c) Latest 300 messages from specific topics (regex) 

waiops-mustgather.sh -P aimanager -Z 'COLLECT_ONLY=kafka##DATA_DUMP=1##AIMGR_KAFKA_SEARCH_SIZE=300##AIMGR_KAFKA_TOPIC_REGEX=topology'

NOTE: The default setting for AIMGR_KAFKA_SEARCH_TOPIC_DIRECTION is "end" (latest), so there is no need to set it during execution.

(d) Custom "kcat" query on Kafka topics

waiops-mustgather.sh -P aimanager -Z 'COLLECT_ONLY=kafka##DATA_DUMP=1##AIMGR_KAFKA_CONSUMER_QUERY="-J -o -5 -e"'

(3) To list Kafka files (partitions, segments, index, etc) on pods

Some problems (eg. Kafka PVCs are filling up fast) would require a check on physical Kafka files on the pods. 
While it is not difficult, it can be tedious. 

waiops-mustgather.sh -Rm kafka-dirlist:mc.cfg

NOTE: The command provides output both on screen and in 0-README-FIRST/SUMMARY.log in the output file.

(4) Everything together!!

waiops-mustgather.sh -ypk pod,configmap,secret -P aimanager -Z 'COLLECT_ONLY=kafka##DATA_DUMP=1##AIMGR_KAFKA_SEARCH_SIZE=all##AIMGR_KAFKA_TOPIC_REGEX=connector' -m kafka-dirlist:mc.cfg

The command above will collect message count and all messages from topics that match regex 'connector' and Kafka file listing on pods.
Besides that, it will also collect pod, configmap and secret data (get, describe, yaml, logs and previous logs) for all product namespaces through the "-ypk pod,configmap,secret" flags.

To learn more about Kafka related parameters, execute the following command and look under [KAFKA] section:

waiops-mustgather.sh -P .listparams

========================================================================================================================

We hope the CP4AIOps MustGather is able to help you to avoid dull repetitive tasks with lengthy commands.

Troubleshooting does not really need to come with all the overhead and pain :)

** Provided cluster-admin privilege, CP4AIOps MustGather is capable of scanning all namespaces to locate the namespace CP4AIOps is installed on. If cluster-admin privilege is unavailable, it will check for default/recommended namespaces. If that fails too, it will prompt the user for namespace details. Of course, the user can always provide namespace details through the '-W AIMGR_NS=xxx' option during execution.