AIOps

 View Only
  • 1.  How to reset Kafka?

    Posted Tue November 23, 2021 05:38 PM
    I have NOI 1.6.3 in a production environment, running 7 pods of kafka. 3 of these pods are not working with a mount issue problem I am trying to solve.

    Currently, I am getting the following errors in the Kafka pods that are running:
    [2021-11-23 20:37:48,838] ERROR [noi-hybrid-shared1-kafka 0] [Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 9 for partition noi-requests-5 (last update controller epoch 9) but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
    [2021-11-23 20:38:21,076] INFO [noi-hybrid-shared1-kafka 0] [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition asm.netcool.events.json-19 as the leader reported an error: UNKNOWN_TOPIC_OR_PARTITION (kafka.server.ReplicaFetcherThread)

    To me, it seems it lost some references (probably due to the mount problem in the other pods) and now it is not working.

    I was wondering if I could restart it fresh, i.e., clean up any current data/messages/references in Kafka and let it start from scratch. Is it possible?

    Any help is really appreciated. Thanks in advance, Fernando.

    ------------------------------
    Fernando Cavalcanti
    ------------------------------


  • 2.  RE: How to reset Kafka?

    User Group Leader
    Posted Wed November 24, 2021 03:04 AM
    Hi Fernando,

    Your issue seems to be uncommon. I'd suggest using IBM support to help as well as asking for their guidance on any reset procedure.

    ------------------------------
    Angus Jamieson
    IT Service Management Solutions Architect
    IBM
    Edinburgh
    ------------------------------



  • 3.  RE: How to reset Kafka?

    Posted Fri November 26, 2021 07:35 AM
      |   view attached
    Just an update on this issue. I ended up not resetting Kafka, but fixing the original issue with kafka. I had 7 kafka pods, but 3 were not working due to PVC issues. The steps I did where:
    - Scale down your kafka to 0 :  $ oc scale statefulset noi-hybrid-shared1-kafka --replicas=0
    - Recreate the PVCs (see yaml attached) : $ oc create -f pvc-kafka.yaml
    - Scale back up : oc scale statefulset noi-hybrid-shared1-kafka --replicas=7

    Then all kafka pods came up.

    ------------------------------
    Fernando Cavalcanti
    ------------------------------

    Attachment(s)

    yaml
    pvc-kafka.yaml   524 B 1 version