Good Morning,
after a fresh install of instana version 255 on a fresh openshift cluster(4.12), I'm getting errors in the oltp-acceptor.instana-core connecting to the instana-kafka-0.instana-kafka pod. (this go's for all 3 of the kafka pods)I have validated that the instana-core config secret contains the correct password for the kafka clusterHas anyone seen something like this? Everything else seems up, no odd loggs in any other pod. the UI is up, and is working, but due to this no agents are reported. The agents are however connecting, and not erroring out. Validated by setting the agent logs to
DEBUG and checking them out.Debug process
# oltp-acceptor.instana-core log line
2023-10-13 11:04:43,121 WARN otlp-acceptor o.apache.kafka.clients.NetworkClient - [Producer clientId=onprem__otlp_acceptor-spans] Bootstrap broker instana-kafka-bootstrap.instana-kafka:9092 (id: -1 rack: null) disconnected
# instana-kafka-0.instana-kafka pod log line
2023-10-13 11:06:33,680 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.131.0.24 (channelId=10.128.2.19:9092-10.131.0.24:48010-103043) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(SCRAM-9092)-SASL_PLAINTEXT-5]
# passwords verified by exporting the strimzi-kafka-user-secret
kubectl get secret strimzi-kafka-user -n instana-kafka --template='{{index .data.password | base64decode}}' && echo
SYbffoSapluG3hakpv9bggC48gAkAimw
# config. instana-core secret
...
kafkaConfig:
adminUser: strimzi-kafka-user
adminPassword: SYbffoSapluG3hakpv9bggC48gAkAimw
consumerUser: strimzi-kafka-user
consumerPassword: SYbffoSapluG3hakpv9bggC48gAkAimw
producerUser: strimzi-kafka-user
producerPassword:
I have validated that SCRAM-SHA512 is enabled
```
kubectl get core instana-core -o=jsonpath='{.spec.datastoreConfigs.kafkaConfig}'
{"authEnabled":true,"hosts":["instana-kafka-bootstrap.instana-kafka"],"replicationFactor":1,"saslMechanism":"SCRAM-SHA-512"}
```
The documentation has the kafka install 0.36 from helm with the actual image 0.35.1. But the install I ran pulled image 0.36. So maybe its a version issue. However deleting kafka and reinstalling with the 35.1 chart had no change. All secrets / passwords were updated during the updare.
I'm sort of scratching my chin at this one.
------------------------------
Chris Watkins
------------------------------