AIOps

 View Only

  • 1.  MessageBus Gateway, kafka and SASL_SSL

    Posted Wed November 09, 2022 11:46 AM
    Hello,
    I am trying to set up a message bus gateway (latest version v14, under Omnibus 8.1.0.28) as a kafka producer to send events to a kafka environment.
    The kafka environment uses SASL_SSL with SCRAM-SHA-512.

    When I follow the documentation and set "security.protocol=SASL_SSL" in kafkaClient.properties, no communication happens at all, and the gateway terminates with (gateway log)
    • Error: E-GJA-000-000: [ngjava]: XMLGateway: Kafka transport initialization error: class org.apache.kafka.common.KafkaException:Failed to construct kafka producer
    • Debug: D-GJA-000-000: [ngjava]: XMLGateway: **************************** CONNECT **********************
    • Error: E-GJA-000-000: [ngjava]: XMLGateway: com.ibm.tivoli.netcool.integrations.transportmodule.TransportConnectException: Initialization fails. Abort connection attempt.
    • at com.ibm.tivoli.netcool.integrations.transportmodule.KafkaTransport(null:-1)
    • at com.ibm.tivoli.netcool.integrations.xml.XMLGateway(XMLGateway.java:293)
    and (transport.log)
    • TRACE [Thread-4] metrics.Metrics (Metrics.java:578) - Registered metric named MetricName [name=buffer-available-bytes, group=producer-metrics, description=The total amount of buffer memory that is not being used (either unallocated or in the free list)., tags={client-id=NetcoolKafkaProducer}]
      TRACE [Thread-4] metrics.Metrics (Metrics.java:421) - Added sensor with name errors
      INFO [Thread-4] producer.KafkaProducer (KafkaProducer.java:1204) - [Producer clientId=NetcoolKafkaProducer] Closing the Kafka producer with timeoutMillis = 0 ms.
      INFO [Thread-4] metrics.Metrics (Metrics.java:659) - Metrics scheduler closed
      INFO [Thread-4] metrics.Metrics (Metrics.java:663) - Closing reporter org.apache.kafka.common.metrics.JmxReporter
      INFO [Thread-4] metrics.Metrics (Metrics.java:669) - Metrics reporters closed
      INFO [Thread-4] utils.AppInfoParser (AppInfoParser.java:83) - App info kafka.producer for NetcoolKafkaProducer unregistered
      DEBUG [Thread-4] producer.KafkaProducer (KafkaProducer.java:1256) - [Producer clientId=NetcoolKafkaProducer] Kafka producer has been closed
    No traffic at all, and no further error messages, although I believe I have activated all possible tracing/debugging.

    When I set "security.protocol=SSL", a communication is established; I can see in the SSL handshake trace that certificates are correct and communication starts. However the kafka broker terminates right when the liveness check starts - which is kind of logical when using the incorrect security protocol.

    Can somebody point me in the direction why almost nothing happens when I configure SASL_SSL ?

    Many thanks in  advance,
    Michael

    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------


  • 2.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Wed November 09, 2022 03:00 PM
    Hi Michael,

    I found this article from 2019 from support that may or may not be of use to you:

    IBM Support

    Here's the text of that link in case the link goes away:

    25 Mar 2019 (4 years ago)
Environment:

IBM Tivoli Netcool/OMNIbus 8.1.0.15

Netcool/OMNIbus Common sslutility 1.1.0.0

Netcool/OMNIbus Common transformer 1.8.0.0

Netcool/OMNIbus Common transportmodule 1.18.0.0

Netcool/OMNIbus Gateway libngjava 1.7.0.0

Netcool/OMNIbus Gateway nco-g-xml 1.9.0.0

IBM Java 1.7

The Message bus gateway is configured to send events to kafka server using SASL_SSL/Kerberos(GSSAPI) mechanism. The gateway fails on making the connection via TLSv1.2 handshake resulting in Kafka transport initialization error

Error: E-GJA-000-000: [ngjava]: XMLGateway: Kafka transport initialization error: Failed to construct kafka producer

Error: E-GJA-000-000: [ngjava]: XMLGateway: com.ibm.tivoli.netcool.integrations.transportmodule.TransportConnectException: Initialization fails. Abort connection attempt. at com.ibm.tivoli.netcool.integrations.transportmodule.KafkaTransport(null:-1) at com.ibm.tivoli.netcool.integrations.xml.XMLGateway(XMLGateway.java:176)

How to reslove this issue ?

Answer:

25 Mar 2019 (4 years ago)
At this stage, we can workaound this issue by configuring the gateway using Oracle Java. The gateway is able to work with Oracle Java when the gateway connects to kafka server using SASL_SSL/Kerberos(GSSAPI) mechanism. Below are the steps to configure the gateway to start using Oracle Java

In kafkaNGOSSclient_jass.conf file, replace ibm with sun

KafkaClient {

com.sun.security.auth.module.Krb5LoginModule required

In $NCHOME/omnibus/platform/linux2x86/bin64/nco_g_xml.env file, modify JRE_DIR pointing to Oracle Java of the directory that contains 'jre' directory, for exmple, if java binary is in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/bin

[root@omnibus bin]# pwd

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/bin

[root@omnibus bin]# ls

java jjs keytool orbd pack200 policytool rmid rmiregistry servertool tnameserv unpack200

You will need set JRE_DIR=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64

JRE_DIR should be set right before if [ "$JRE_DIR" = "" -o ! -x "$JRE_DIR/jre/bin/java" ], like below

JRE_DIR=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 if [ "$JRE_DIR" = "" -o ! -x "$JRE_DIR/jre/bin/java" ] then echo Cannot find your Java environment exit 1 fi
Set LD_LIBRARY_PATH pointing to Oracle Java. LD_LIBRARY_PATH should be pointed to the directory containing  http://libjvm.so  file, for example,  http://libjvm.so  is in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/lib/amd64/server

[root@omnibus server]# pwd

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/lib/amd64/server

[root@omnibus server]# ls

classes.jsa  http://libjsig.so  http://libjvm.so  Xusage.txt

Set LD_LIBRARY_PATH to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre/lib/amd64/server

Now the gateway is configured to use Oracle Java.​


    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us help with any difficult IT problems.
    ------------------------------

    Original Message


  • 3.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Thu November 10, 2022 03:09 AM
    Hi Frank,
    many thanks for pointing me there. Certainly worth an attempt - I'll post my experience here then.
    Best regards,
    Michael

    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------

    Original Message


  • 4.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Wed November 16, 2022 02:29 AM

    ok, as promised - here is the reason.
    It was not the JAVA version - moving to Oracle Java didn't help in this case.
    It was just a missing Semicolon.
    Many thanks to Jim Hutchinson from support who pointed me to that syntax error in the jaas_conf file.

    Unfortunately, the SASL APIs remains very silent when there is a syntax error in the config. It just doesn't do anything ....

    So, when I work with kafka and SASL in the future, I better scrutinize my config files threeeee than just two times.



    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------

    Original Message


  • 5.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Wed August 14, 2024 03:11 PM

    Hi EveryOne,

    I'm getting this error while i'm starting the Messagebus probe.

    Consumer property 'pollInterval' is not configured. Use default value: 1000 ms.
    2024-08-14T10:00:06: Debug: D-JPR-000-000: [KafkaConsumerRunnable] - Instantiating Kafka Consumer - Started
    2024-08-14T10:00:06: Error: E-JPR-000-000: Kafka transport initialization error: class org.apache.kafka.common.KafkaException:Failed to construct kafka consumer
    2024-08-14T10:00:06: Debug: D-JPR-000-000: ***********************************************************
    2024-08-14T10:00:06: Debug: D-JPR-000-000: **************************** CONNECT **********************
    2024-08-14T10:00:06: Debug: D-JPR-000-000: ***********************************************************
    2024-08-14T10:00:06: Error: E-JPR-000-000: Fail to start Transport module for connection
    2024-08-14T10:00:06: Error: E-JPR-000-000: Failed to connect; ProbeException: Fail to start Transport module for connection; TransportConnectException: Initialization fails. Abort connection attempt.
    2024-08-14T10:00:06: Information: I-UNK-000-000: Probewatch: Unable to get events. Failed to connect; ProbeException: Fail to start Transport module for connection; TransportConnectException: Initialization fails. Abort connection attempt.
    2024-08-14T10:00:06: Debug: D-UNK-000-000: Rules file processing took 14 usec.
    2024-08-14T10:00:06: Information: I-UNK-000-000: Event processed by rules. discard=false
    2024-08-14T10:00:06: Debug: D-UNK-000-000: Write 3710 bytes to raw capture file.
    2024-08-14T10:00:06: Debug: D-UNK-000-000: Flushing events to object servers
    2024-08-14T10:00:06: Debug: D-UNK-000-000: 1 buffered alerts
    2024-08-14T10:00:06: Debug: D-JPR-000-000: com.ibm.tivoli.netcool.omnibus.probe.ProbeException: Fail to start Transport module for connection
            at com.ibm.tivoli.oidk.ProbeImpl.connect(Unknown Source)
            at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.connect(Unknown Source)
            at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.connectAndRun(Unknown Source)
            at com.ibm.tivoli.netcool.omnibus.probe.framework.ProbeRunner.run(Unknown Source)
            at com.ibm.tivoli.netcool.omnibus.oidk.Probe.start(Unknown Source)
            at com.ibm.tivoli.netcool.omnibus.oidk.Probe.main(Unknown Source)
    Caused by: com.ibm.tivoli.netcool.integrations.transportmodule.TransportConnectException: Initialization fails. Abort connection attempt.
            at com.ibm.tivoli.netcool.integrations.transportmodule.KafkaTransport.connect(Unknown Source)
            ... 6 more



    ------------------------------
    Udaya Durga Polabathula
    ------------------------------

    Original Message


  • 6.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Thu August 15, 2024 01:40 AM

    Hi Udaya,

    did you check out the latest versions Jim Hutchinson's out the support guides for the Message Bus Probe (and Gateway) ? There is a special one for kafka transport now.

    They can be found here:

    https://www.ibm.com/support/pages/supports-guide-message-bus-integration-ncogxml-and-ncopmessagebus

    Hope that helps

    Michael



    ------------------------------
    Michael Troitzsch
    Solution Architect
    DICOS GmbH
    Darmstadt
    ------------------------------

    Original Message


  • 7.  RE: MessageBus Gateway, kafka and SASL_SSL

    Posted Thu November 10, 2022 02:54 PM
    Thanks Frank for posting the text as well, as some of those links will be changing in a couple of weeks.

    ------------------------------
    Stephanie Wilkerson
    IBM
    ------------------------------

    Original Message