I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?

Batching size is only applicable when auto commit is off. The FSH will grab all messages available.  Batching is just a measure of how many messages can be committed together at same time.  

I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?

I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?

The Kafka Handler batch size is only used when the cluster auto commit is off.   So, when auto commit is on, it should process all available messages without respect to batch size of the handler.

The question is, what is your batch size in your handler, and, are that many messages available for consumption?

I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?

The batch size is set to 20, and we are publishing 105 messages to the topic. However, it is consuming all the messages, bypassing the batch size.

• Config 1 – Auto commit enabled, batch size = 1 → Consumes all messages.

• Config 2 – Auto commit enabled, batch size = 20 → Consumes all 105 messages.

• Config 3 – Auto commit disabled, batch size = 1 or 20 → Does not consume any messages.
  
  As per documentation batch-size will only work when auto commit is off but when we are making it off datapower does not consume any message from Topic.
  As soon as we enable auto commit it consumes all the message in one go.

The Kafka Handler batch size is only used when the cluster auto commit is off.   So, when auto commit is on, it should process all available messages without respect to batch size of the handler.

The question is, what is your batch size in your handler, and, are that many messages available for consumption?

I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?

• Config 1 – Auto commit enabled, batch size = 1 → Consumes all messages.   <-- This is what it should do, batch size does not matter

• Config 2 – Auto commit enabled, batch size = 20 → Consumes all 105 messages.     <-- This is what it should do, batch size does not matter

• Config 3 – Auto commit disabled, batch size = 1 or 20 → Does not consume any messages.   <-- This is the mystery.

Do you happen to have anything in the "property" property of the cluster?   I'm thinking you may have set the "auto.commit.interval.ms", and, maybe just maybe, if auto commit is disabled and that property is set, maybe it breaks it??   I really don't know, but, just thinking out loud.

The batch size is set to 20, and we are publishing 105 messages to the topic. However, it is consuming all the messages, bypassing the batch size.

• Config 1 – Auto commit enabled, batch size = 1 → Consumes all messages.

• Config 2 – Auto commit enabled, batch size = 20 → Consumes all 105 messages.

• Config 3 – Auto commit disabled, batch size = 1 or 20 → Does not consume any messages.
  
  As per documentation batch-size will only work when auto commit is off but when we are making it off datapower does not consume any message from Topic.
  As soon as we enable auto commit it consumes all the message in one go.

The Kafka Handler batch size is only used when the cluster auto commit is off.   So, when auto commit is on, it should process all available messages without respect to batch size of the handler.

The question is, what is your batch size in your handler, and, are that many messages available for consumption?

I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?

We did not set this property in the cluster properties. However, we have tried multiple properties to make it work, but nothing has worked out so far. Our goal is to achieve 20 Transactions Per Second (TPS), and we are initially trying to achieve this by adjusting the batch size. Unfortunately, adjusting the batch size is not having the desired effect. Additionally, we cannot use Service Level Monitoring (SLM) as it may lead to a loss of messages, which is not acceptable in our use case.

• Config 1 – Auto commit enabled, batch size = 1 → Consumes all messages.   <-- This is what it should do, batch size does not matter

• Config 2 – Auto commit enabled, batch size = 20 → Consumes all 105 messages.     <-- This is what it should do, batch size does not matter

• Config 3 – Auto commit disabled, batch size = 1 or 20 → Does not consume any messages.   <-- This is the mystery.

Do you happen to have anything in the "property" property of the cluster?   I'm thinking you may have set the "auto.commit.interval.ms", and, maybe just maybe, if auto commit is disabled and that property is set, maybe it breaks it??   I really don't know, but, just thinking out loud.

The batch size is set to 20, and we are publishing 105 messages to the topic. However, it is consuming all the messages, bypassing the batch size.

• Config 1 – Auto commit enabled, batch size = 1 → Consumes all messages.

• Config 2 – Auto commit enabled, batch size = 20 → Consumes all 105 messages.

• Config 3 – Auto commit disabled, batch size = 1 or 20 → Does not consume any messages.
  
  As per documentation batch-size will only work when auto commit is off but when we are making it off datapower does not consume any message from Topic.
  As soon as we enable auto commit it consumes all the message in one go.

The Kafka Handler batch size is only used when the cluster auto commit is off.   So, when auto commit is on, it should process all available messages without respect to batch size of the handler.

The question is, what is your batch size in your handler, and, are that many messages available for consumption?

I'm working on Kafka integration in IBM DataPower and facing an issue with batch message consumption.

When I set auto commit to off, the Kafka Handler does not consume any messages at all.
However, when I set auto commit to on, DataPower processes all available messages instead of the batch size defined in the Kafka Handler configuration.

Has anyone encountered this behavior? How can I configure DataPower so that it picks messages in the specified batch size without consuming all messages at once?