AIOps

 View Only
  • 1.  MessageBus Gateway, kafka and SASL_SSL

    Posted Wed November 09, 2022 11:46 AM
    Hello,
    I am trying to set up a message bus gateway (latest version v14, under Omnibus 8.1.0.28) as a kafka producer to send events to a kafka environment.
    The kafka environment uses SASL_SSL with SCRAM-SHA-512.

    When I follow the documentation and set "security.protocol=SASL_SSL" in kafkaClient.properties, no communication happens at all, and the gateway terminates with (gateway log)
    • Error: E-GJA-000-000: [ngjava]: XMLGateway: Kafka transport initialization error: class org.apache.kafka.common.KafkaException:Failed to construct kafka producer
    • Debug: D-GJA-000-000: [ngjava]: XMLGateway: **************************** CONNECT **********************
    • Error: E-GJA-000-000: [ngjava]: XMLGateway: com.ibm.tivoli.netcool.integrations.transportmodule.TransportConnectException: Initialization fails. Abort connection attempt.
    • at com.ibm.tivoli.netcool.integrations.transportmodule.KafkaTransport(null:-1)
    • at com.ibm.tivoli.netcool.integrations.xml.XMLGateway(XMLGateway.java:293)
    and (transport.log)
    • TRACE [Thread-4] metrics.Metrics (Metrics.java:578) - Registered metric named MetricName [name=buffer-available-bytes, group=producer-metrics, description=The total amount of buffer memory that is not being used (either unallocated or in the free list)., tags={client-id=NetcoolKafkaProducer}]
      TRACE [Thread-4] metrics.Metrics (Metrics.java:421) - Added sensor with name errors
      INFO [Thread-4] producer.KafkaProducer (KafkaProducer.java:1204) - [Producer clientId=NetcoolKafkaProducer] Closing the Kafka producer with timeoutMillis = 0 ms.
      INFO [Thread-4] metrics.Metrics (Metrics.java:659) - Metrics scheduler closed
      INFO [Thread-4] metrics.Metrics (Metrics.java:663) - Closing reporter org.apache.kafka.common.metrics.JmxReporter
      INFO [Thread-4] metrics.Metrics (Metrics.java:669) - Metrics reporters closed
      INFO [Thread-4] utils.AppInfoParser (AppInfoParser.java:83) - App info kafka.producer for NetcoolKafkaProducer unregistered
      DEBUG [Thread-4] producer.KafkaProducer (KafkaProducer.java:1256) - [Producer clientId=NetcoolKafkaProducer] Kafka producer has been closed
    No traffic at all, and no further error messages, although I believe I have activated all possible tracing/debugging.

    When I set "security.protocol=SSL", a communication is established; I can see in the SSL handshake trace that certificates are correct and communication starts. However the kafka broker terminates right when the liveness check starts - which is kind of logical when using the incorrect security protocol.

    Can somebody point me in the direction why almost nothing happens when I configure SASL_SSL ?

    Many thanks in  advance,
    Michael

    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------


  • 2.  RE: MessageBus Gateway, kafka and SASL_SSL

    InnerCircle
    Posted Wed November 09, 2022 03:00 PM
    Hi Michael,

    I found this article from 2019 from support that may or may not be of use to you:

    IBM Support

    Here's the text of that link in case the link goes away:

    25 Mar 2019 (4 years ago)
    Environment:
    
    IBM Tivoli Netcool/OMNIbus 8.1.0.15
    
    Netcool/OMNIbus Common sslutility 1.1.0.0
    
    Netcool/OMNIbus Common transformer 1.8.0.0
    
    Netcool/OMNIbus Common transportmodule 1.18.0.0
    
    Netcool/OMNIbus Gateway libngjava 1.7.0.0
    
    Netcool/OMNIbus Gateway nco-g-xml 1.9.0.0
    
    IBM Java 1.7
    
    The Message bus gateway is configured to send events to kafka server using SASL_SSL/Kerberos(GSSAPI) mechanism. The gateway fails on making the connection via TLSv1.2 handshake resulting in Kafka transport initialization error
    
    Error: E-GJA-000-000: [ngjava]: XMLGateway: Kafka transport initialization error: Failed to construct kafka producer
    
    Error: E-GJA-000-000: [ngjava]: XMLGateway: com.ibm.tivoli.netcool.integrations.transportmodule.TransportConnectException: Initialization fails. Abort connection attempt. at com.ibm.tivoli.netcool.integrations.transportmodule.KafkaTransport(null:-1) at com.ibm.tivoli.netcool.integrations.xml.XMLGateway(XMLGateway.java:176)
    
    How to reslove this issue ?
    
    Answer:
    
    25 Mar 2019 (4 years ago)
    At this stage, we can workaound this issue by configuring the gateway using Oracle Java. The gateway is able to work with Oracle Java when the gateway connects to kafka server using SASL_SSL/Kerberos(GSSAPI) mechanism. Below are the steps to configure the gateway to start using Oracle Java
    
    In kafkaNGOSSclient_jass.conf file, replace ibm with sun
    
    KafkaClient {
    
    com.sun.security.auth.module.Krb5LoginModule required
    
    In $NCHOME/omnibus/platform/linux2x86/bin64/nco_g_xml.env file, modify JRE_DIR pointing to Oracle Java of the directory that contains 'jre' directory, for exmple, if java binary is in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/bin
    
    [root@omnibus bin]# pwd
    
    /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/bin
    
    [root@omnibus bin]# ls
    
    java jjs keytool orbd pack200 policytool rmid rmiregistry servertool tnameserv unpack200
    
    You will need set JRE_DIR=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
    
    JRE_DIR should be set right before if [ "$JRE_DIR" = "" -o ! -x "$JRE_DIR/jre/bin/java" ], like below
    
    JRE_DIR=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 if [ "$JRE_DIR" = "" -o ! -x "$JRE_DIR/jre/bin/java" ] then echo Cannot find your Java environment exit 1 fi
    Set LD_LIBRARY_PATH pointing to Oracle Java. LD_LIBRARY_PATH should be pointed to the directory containing  http://libjvm.so  file, for example,  http://libjvm.so  is in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/lib/amd64/server
    
    [root@omnibus server]# pwd
    
    /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/lib/amd64/server
    
    [root@omnibus server]# ls
    
    classes.jsa  http://libjsig.so  http://libjvm.so  Xusage.txt
    
    Set LD_LIBRARY_PATH to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/lib/amd64/server
    
    Now the gateway is configured to use Oracle Java.​


    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us help with any difficult IT problems.
    ------------------------------



  • 3.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Thu November 10, 2022 03:09 AM
    Hi Frank,
    many thanks for pointing me there. Certainly worth an attempt - I'll post my experience here then.
    Best regards,
    Michael

    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------



  • 4.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Wed November 16, 2022 02:29 AM

    ok, as promised - here is the reason.
    It was not the JAVA version - moving to Oracle Java didn't help in this case.
    It was just a missing Semicolon.
    Many thanks to Jim Hutchinson from support who pointed me to that syntax error in the jaas_conf file.

    Unfortunately, the SASL APIs remains very silent when there is a syntax error in the config. It just doesn't do anything ....

    So, when I work with kafka and SASL in the future, I better scrutinize my config files threeeee than just two times.



    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------



  • 5.  RE: MessageBus Gateway, kafka and SASL_SSL

    Community Leadership
    Posted Thu November 10, 2022 02:54 PM
    Thanks Frank for posting the text as well, as some of those links will be changing in a couple of weeks.

    ------------------------------
    Stephanie Wilkerson
    IBM
    ------------------------------