Platform and Technology Integration

 View Only

  • 1.  OCP Install, Kafka failure

    Posted Fri October 13, 2023 11:01 AM
    Good Morning,
    after a fresh install of instana version 255 on a fresh openshift cluster(4.12), I'm getting errors in the oltp-acceptor.instana-core connecting to the instana-kafka-0.instana-kafka pod. (this go's for all 3 of the kafka pods)I have validated that the instana-core config secret contains the correct password for the kafka clusterHas anyone seen something like this? Everything else seems up, no odd loggs in any other pod. the UI is up, and is working, but due to this no agents are reported. The agents are however connecting, and not erroring out. Validated by setting the agent logs to
    DEBUG and checking them out.Debug process
    # oltp-acceptor.instana-core log line 
2023-10-13 11:04:43,121 WARN otlp-acceptor o.apache.kafka.clients.NetworkClient - [Producer clientId=onprem__otlp_acceptor-spans] Bootstrap broker instana-kafka-bootstrap.instana-kafka:9092 (id: -1 rack: null) disconnected

# instana-kafka-0.instana-kafka pod log line 
2023-10-13 11:06:33,680 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.131.0.24 (channelId=10.128.2.19:9092-10.131.0.24:48010-103043) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(SCRAM-9092)-SASL_PLAINTEXT-5]

# passwords verified by  exporting the strimzi-kafka-user-secret
kubectl get secret strimzi-kafka-user  -n instana-kafka --template='{{index .data.password | base64decode}}' && echo

SYbffoSapluG3hakpv9bggC48gAkAimw

# config. instana-core secret
...
 kafkaConfig:
    adminUser: strimzi-kafka-user
    adminPassword: SYbffoSapluG3hakpv9bggC48gAkAimw
    consumerUser: strimzi-kafka-user
    consumerPassword: SYbffoSapluG3hakpv9bggC48gAkAimw
    producerUser: strimzi-kafka-user
    producerPassword:

    I have validated that SCRAM-SHA512 is enabled 
    ```

    kubectl get core instana-core -o=jsonpath='{.spec.datastoreConfigs.kafkaConfig}'
    {"authEnabled":true,"hosts":["instana-kafka-bootstrap.instana-kafka"],"replicationFactor":1,"saslMechanism":"SCRAM-SHA-512"}

    ```



    ------------------------------
    Chris Watkins
    ------------------------------


  • 2.  RE: OCP Install, Kafka failure

    Posted Sun October 15, 2023 04:37 AM

    Chris I had the same problem and found it makes a difference in the network interfaces I Had two interfaces a public and private active, and did several scenarios with one interface down the same problem. Finally, I did an air-gapped installation and this was working

    Regards Tjeerd



    ------------------------------
    Tjeerd Saijoen
    CEO
    RufusAI
    Chicago
    3124665579
    ------------------------------

    Original Message


  • 3.  RE: OCP Install, Kafka failure

    Posted Tue October 17, 2023 11:30 AM

    Hi Chris,

    thanks for your question. We were made aware of this wrong behavior by other support cases, and this is indeed a bug. Please use versions 257-3 and 259-0 which should fix the issue. There is currently no fix for 255. In the future, feel free to open a support case.

    Best



    ------------------------------
    Hubert Hesse
    Senior Product Manager @Instana
    Self-Hosting
    ------------------------------

    Original Message


  • 4.  RE: OCP Install, Kafka failure

    Posted Wed October 18, 2023 09:06 AM

    Thank you Hubert,

    I will reinstall with the latest 259-0.

    Thanks!



    ------------------------------
    Chris Watkins
    ------------------------------

    Original Message