App Connect

Is there a way in IBM App Connect Enterprise 12.0.4.0 to programmatically commit the offset in Kafka?

By enabling the "Commit message offset in Kafka" property on the Kafka Consumer node, the offset is always and "properly" committed; however, even in case of an exception during flow execution, the offset is committed and this is not the desired behavior; in case of an exception I would prefer to perform remediation operations before committing the offset and losing the message.

Consider the following use case:

The main flow consists of a:
* a KafkaConsumer node, reading from a remote kafka topic.
* a Mapping node, which performs a JSON to DFDL (expressed by a Copy Cobol) conversion of the message.
* a MQOutput node, to write to a remote IBM MQ server.

 If the Mapping node generates an exception (because the read message is not cobol copy compliant), I would like to make sure that the KafkaConsumer does not commit the offset (and possibly be looped) and I don't "lose" the message.

Or, if the MQOutput node raises an exception (MQ Remote Server Unreachable, MQ Queue Full, etc.), again, I would like to make sure that the KafkaConsumer does not commit the offset (and possibly be in a loop) and I don't "lose" the message.

Can someone help me understand how to properly handle the message offset in case of an error during flow execution?

Reposting this query.. I also have similar requirement of manually committing the consumer offset depending on the execution of the message flow. Can we manually set consumer message to offset commit?

Original Message:
Sent: Wed July 26, 2023 11:48 AM
From: Paolo Alberti
Subject: Kafka consumer - programmatically commit offset in Kafka in case of error.

Is there a way in IBM App Connect Enterprise 12.0.4.0 to programmatically commit the offset in Kafka?

By enabling the "Commit message offset in Kafka" property on the Kafka Consumer node, the offset is always and "properly" committed; however, even in case of an exception during flow execution, the offset is committed and this is not the desired behavior; in case of an exception I would prefer to perform remediation operations before committing the offset and losing the message.

Consider the following use case:

The main flow consists of a:
* a KafkaConsumer node, reading from a remote kafka topic.
* a Mapping node, which performs a JSON to DFDL (expressed by a Copy Cobol) conversion of the message.
* a MQOutput node, to write to a remote IBM MQ server.

 If the Mapping node generates an exception (because the read message is not cobol copy compliant), I would like to make sure that the KafkaConsumer does not commit the offset (and possibly be looped) and I don't "lose" the message.

Or, if the MQOutput node raises an exception (MQ Remote Server Unreachable, MQ Queue Full, etc.), again, I would like to make sure that the KafkaConsumer does not commit the offset (and possibly be in a loop) and I don't "lose" the message.

Can someone help me understand how to properly handle the message offset in case of an error during flow execution?

------------------------------
Paolo Alberti
------------------------------