I have NOI 1.6.3 in a production environment, running 7 pods of kafka. 3 of these pods are not working with a mount issue problem I am trying to solve.

Currently, I am getting the following errors in the Kafka pods that are running:
[2021-11-23 20:37:48,838] ERROR [noi-hybrid-shared1-kafka 0] [Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 9 for partition noi-requests-5 (last update controller epoch 9) but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
[2021-11-23 20:38:21,076] INFO [noi-hybrid-shared1-kafka 0] [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition asm.netcool.events.json-19 as the leader reported an error: UNKNOWN_TOPIC_OR_PARTITION (kafka.server.ReplicaFetcherThread)

To me, it seems it lost some references (probably due to the mount problem in the other pods) and now it is not working.

I was wondering if I could restart it fresh, i.e., clean up any current data/messages/references in Kafka and let it start from scratch. Is it possible?

Any help is really appreciated. Thanks in advance, Fernando.

------------------------------
Fernando Cavalcanti
------------------------------

Hi Fernando,

Your issue seems to be uncommon. I'd suggest using IBM support to help as well as asking for their guidance on any reset procedure.

------------------------------
Angus Jamieson
IT Service Management Solutions Architect
IBM
Edinburgh
------------------------------

Original Message:
Sent: Tue November 23, 2021 04:32 PM
From: Fernando Cavalcanti
Subject: How to reset Kafka?

I have NOI 1.6.3 in a production environment, running 7 pods of kafka. 3 of these pods are not working with a mount issue problem I am trying to solve.

Currently, I am getting the following errors in the Kafka pods that are running:
[2021-11-23 20:37:48,838] ERROR [noi-hybrid-shared1-kafka 0] [Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 9 for partition noi-requests-5 (last update controller epoch 9) but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
[2021-11-23 20:38:21,076] INFO [noi-hybrid-shared1-kafka 0] [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition asm.netcool.events.json-19 as the leader reported an error: UNKNOWN_TOPIC_OR_PARTITION (kafka.server.ReplicaFetcherThread)

To me, it seems it lost some references (probably due to the mount problem in the other pods) and now it is not working.

I was wondering if I could restart it fresh, i.e., clean up any current data/messages/references in Kafka and let it start from scratch. Is it possible?

Any help is really appreciated. Thanks in advance, Fernando.

------------------------------
Fernando Cavalcanti
------------------------------

Just an update on this issue. I ended up not resetting Kafka, but fixing the original issue with kafka. I had 7 kafka pods, but 3 were not working due to PVC issues. The steps I did where:

Then all kafka pods came up.

------------------------------
Fernando Cavalcanti
------------------------------

Original Message:
Sent: Tue November 23, 2021 04:32 PM
From: Fernando Cavalcanti
Subject: How to reset Kafka?

I have NOI 1.6.3 in a production environment, running 7 pods of kafka. 3 of these pods are not working with a mount issue problem I am trying to solve.

Currently, I am getting the following errors in the Kafka pods that are running:
[2021-11-23 20:37:48,838] ERROR [noi-hybrid-shared1-kafka 0] [Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 9 for partition noi-requests-5 (last update controller epoch 9) but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
[2021-11-23 20:38:21,076] INFO [noi-hybrid-shared1-kafka 0] [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition asm.netcool.events.json-19 as the leader reported an error: UNKNOWN_TOPIC_OR_PARTITION (kafka.server.ReplicaFetcherThread)

To me, it seems it lost some references (probably due to the mount problem in the other pods) and now it is not working.

I was wondering if I could restart it fresh, i.e., clean up any current data/messages/references in Kafka and let it start from scratch. Is it possible?

Any help is really appreciated. Thanks in advance, Fernando.

------------------------------
Fernando Cavalcanti
------------------------------