Platform and Technology Integration

 View Only
  • 1.  OCP Install, Kafka failure

    Posted Fri October 13, 2023 11:01 AM
    Good Morning,
    after a fresh install of instana version 255 on a fresh openshift cluster(4.12), I'm getting errors in the oltp-acceptor.instana-core connecting to the instana-kafka-0.instana-kafka pod. (this go's for all 3 of the kafka pods)I have validated that the instana-core config secret contains the correct password for the kafka clusterHas anyone seen something like this? Everything else seems up, no odd loggs in any other pod. the UI is up, and is working, but due to this no agents are reported. The agents are however connecting, and not erroring out. Validated by setting the agent logs to
    DEBUG and checking them out.Debug process
    # oltp-acceptor.instana-core log line 
    2023-10-13 11:04:43,121 WARN otlp-acceptor o.apache.kafka.clients.NetworkClient - [Producer clientId=onprem__otlp_acceptor-spans] Bootstrap broker instana-kafka-bootstrap.instana-kafka:9092 (id: -1 rack: null) disconnected
    # instana-kafka-0.instana-kafka pod log line 
    2023-10-13 11:06:33,680 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with / (channelId= (Unexpected Kafka request of type METADATA during SASL handshake.) ( [data-plane-kafka-network-thread-0-ListenerName(SCRAM-9092)-SASL_PLAINTEXT-5]
    # passwords verified by  exporting the strimzi-kafka-user-secret
    kubectl get secret strimzi-kafka-user  -n instana-kafka --template='{{index .data.password | base64decode}}' && echo
    # config. instana-core secret
        adminUser: strimzi-kafka-user
        adminPassword: SYbffoSapluG3hakpv9bggC48gAkAimw
        consumerUser: strimzi-kafka-user
        consumerPassword: SYbffoSapluG3hakpv9bggC48gAkAimw
        producerUser: strimzi-kafka-user

    I have validated that SCRAM-SHA512 is enabled 

    kubectl get core instana-core -o=jsonpath='{.spec.datastoreConfigs.kafkaConfig}'


    Chris Watkins

  • 2.  RE: OCP Install, Kafka failure

    IBM Champion
    Posted Sun October 15, 2023 04:37 AM

    Chris I had the same problem and found it makes a difference in the network interfaces I Had two interfaces a public and private active, and did several scenarios with one interface down the same problem. Finally, I did an air-gapped installation and this was working

    Regards Tjeerd

    Tjeerd Saijoen

  • 3.  RE: OCP Install, Kafka failure

    Posted Tue October 17, 2023 11:30 AM

    Hi Chris,

    thanks for your question. We were made aware of this wrong behavior by other support cases, and this is indeed a bug. Please use versions 257-3 and 259-0 which should fix the issue. There is currently no fix for 255. In the future, feel free to open a support case.


    Hubert Hesse
    Senior Product Manager @Instana

  • 4.  RE: OCP Install, Kafka failure

    Posted Wed October 18, 2023 09:06 AM

    Thank you Hubert,

    I will reinstall with the latest 259-0.


    Chris Watkins