IBM Event Streams and IBM Event Automation

IBM Event Streams and IBM Event Automation

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

How to use kafka-console-consumer.sh to view the contents of Apache Avro-encoded events

By Dale Lane posted 23 days ago

  
kafka-console-consumer.sh is one of the most useful tools in the Kafka user’s toolkit. But if your topic has Avro-encoded events, the output can be a bit hard to read.
 
 
You don’t have to put up with that, as the tool has a formatter plugin framework. With the right plugin, you can get nicely formatted output from your Avro-encoded events.
 
 
With this in mind, I’ve written a new Avro formatter for a few common Avro situations. You can find it at:
 
 
The README includes instructions on how to add it to your Kafka console command, and configure it with how to find your schema.
 
I won’t duplicate those instructions here, but I do want to briefly call out a few ways you can use it, depending on where your Avro schema is.
 

Schemas in Apicurio Schema Registry

If you keep your schema in an Apicurio schema registry the formatter can dynamically fetch the schemas it needs from the registry (just like your regular deserializer would).
 
 

Schemas in local files

If you keep your schema together with your application source code, and have it available in a local file, then the formatter can just read that local file directly.
 
 

Schemas in Event Endpoint Management

If you’re consuming events from someone else’s topic that you found in Event Endpoint Management, then the formatter can fetch any schemas it needs from the Catalog.
 
 
I hope you’ll find this useful. 
If you have any ideas for what else it could do, please feel free to create an issue in the Github repository.
 

Extra tips

Finally, I want to highlight a couple of quick tips for how to get more out of the formatter:
 

Use a properties file and a shell alias

Save yourself some typing!
 
Instead of typing your schema registry URL and credentials in --property options every time you use kafka-console-consumer.sh, leave them all in a properties file and create an Avro-themed alias for kafka-console-consumer that points at your properties file.
 
 

Pair it with jq

The formatter outputs the parsed Avro-encoded events as a JSON string. This means that you can pipe it to something like jq for colour formatting and advanced parsing and processing.
 
This is very useful with large and complex event data, where you just want to view specific fields.
 
0 comments
12 views

Permalink