AIOps

  • 1.  How to reset Kafka?

    Posted 12 days ago
    I have NOI 1.6.3 in a production environment, running 7 pods of kafka. 3 of these pods are not working with a mount issue problem I am trying to solve.

    Currently, I am getting the following errors in the Kafka pods that are running:
    [2021-11-23 20:37:48,838] ERROR [noi-hybrid-shared1-kafka 0] [Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 9 for partition noi-requests-5 (last update controller epoch 9) but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
    [2021-11-23 20:38:21,076] INFO [noi-hybrid-shared1-kafka 0] [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition asm.netcool.events.json-19 as the leader reported an error: UNKNOWN_TOPIC_OR_PARTITION (kafka.server.ReplicaFetcherThread)

    To me, it seems it lost some references (probably due to the mount problem in the other pods) and now it is not working.

    I was wondering if I could restart it fresh, i.e., clean up any current data/messages/references in Kafka and let it start from scratch. Is it possible?

    Any help is really appreciated. Thanks in advance, Fernando.

    ------------------------------
    Fernando Cavalcanti
    ------------------------------


  • 2.  RE: How to reset Kafka?

    User Group Leader
    Posted 11 days ago
    Hi Fernando,

    Your issue seems to be uncommon. I'd suggest using IBM support to help as well as asking for their guidance on any reset procedure.

    ------------------------------
    Angus Jamieson
    IT Service Management Solutions Architect
    IBM
    Edinburgh
    ------------------------------



  • 3.  RE: How to reset Kafka?

    Posted 9 days ago
      |   view attached
    Just an update on this issue. I ended up not resetting Kafka, but fixing the original issue with kafka. I had 7 kafka pods, but 3 were not working due to PVC issues. The steps I did where:
    - Scale down your kafka to 0 :  $ oc scale statefulset noi-hybrid-shared1-kafka --replicas=0
    - Recreate the PVCs (see yaml attached) : $ oc create -f pvc-kafka.yaml
    - Scale back up : oc scale statefulset noi-hybrid-shared1-kafka --replicas=7

    Then all kafka pods came up.

    ------------------------------
    Fernando Cavalcanti
    ------------------------------

    Attachment(s)

    yaml
    pvc-kafka.yaml   524 B 1 version