View Only
  • 1.  How to reset Kafka?

    Posted Tue November 23, 2021 05:38 PM
    I have NOI 1.6.3 in a production environment, running 7 pods of kafka. 3 of these pods are not working with a mount issue problem I am trying to solve.

    Currently, I am getting the following errors in the Kafka pods that are running:
    [2021-11-23 20:37:48,838] ERROR [noi-hybrid-shared1-kafka 0] [Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 9 for partition noi-requests-5 (last update controller epoch 9) but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
    [2021-11-23 20:38:21,076] INFO [noi-hybrid-shared1-kafka 0] [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition as the leader reported an error: UNKNOWN_TOPIC_OR_PARTITION (kafka.server.ReplicaFetcherThread)

    To me, it seems it lost some references (probably due to the mount problem in the other pods) and now it is not working.

    I was wondering if I could restart it fresh, i.e., clean up any current data/messages/references in Kafka and let it start from scratch. Is it possible?

    Any help is really appreciated. Thanks in advance, Fernando.

    Fernando Cavalcanti

  • 2.  RE: How to reset Kafka?

    User Group Leader
    Posted Wed November 24, 2021 03:04 AM
    Hi Fernando,

    Your issue seems to be uncommon. I'd suggest using IBM support to help as well as asking for their guidance on any reset procedure.

    Angus Jamieson
    IT Service Management Solutions Architect

  • 3.  RE: How to reset Kafka?

    Posted Fri November 26, 2021 07:35 AM
      |   view attached
    Just an update on this issue. I ended up not resetting Kafka, but fixing the original issue with kafka. I had 7 kafka pods, but 3 were not working due to PVC issues. The steps I did where:
    - Scale down your kafka to 0 :  $ oc scale statefulset noi-hybrid-shared1-kafka --replicas=0
    - Recreate the PVCs (see yaml attached) : $ oc create -f pvc-kafka.yaml
    - Scale back up : oc scale statefulset noi-hybrid-shared1-kafka --replicas=7

    Then all kafka pods came up.

    Fernando Cavalcanti


    pvc-kafka.yaml   524 B 1 version