Message Image  

Configuring Kafka with Kerberos Authentication in IBM Integration Bus V10 and App Connect Enterprise V11

 View Only
Thu July 23, 2020 11:16 AM

This article aims at providing a tool (a standalone Java Program) to simplify setting up Kerberos authentication with Kafka nodes. The tool enables you to create a setup and test it outside of the IIB/ACE environment and once you have it working, then to adopt the same configurations to IIB/ACE.

Pre-reqs:

Instructions to setup and run the tool outside of the IIB/ACE environment.

    1. Download the IIBKafkaPubTestV2.jar

      IIBKafkaPubTestV2.jar

    2. Create a standalone Kafka client properties file with the name “producer.properties” and configure the options below based on your Kafka Server settings

    1. Execute the IIBKafkaPubTestV2.jar with below syntax and collect the result.
      Java -Djavax.net.ssl.trustStore=<path> - Djavax.net.ssl.trustStorePassword=<trust_password> -Djavax.net.ssl.keyStore=<path> - Djavax.net.ssl.keyStorePassword=<keystore_password> -jar IIBKafkaPubTestV2.jar

  1. Check kafkalog.txt logs to identify the status of Kafka Publish using the Kerberos authentication. On successful publish you will see the below message

    “successfully published”

    If it throws any error related to SSL Handshake, Kerberos Authentication or any other errors related to Kafka with Kerberos setup then fix the issue until you receive “successfully published” from the IIBKafkaPubTestV2.jar execution.

    Once you have “successfully published” then you are ready to incorporate the setup in to your IIB configuration.

Instructions for configuring Kafka with Kerberos authentication in IIB v10.

Once you have a successful run of the tool, use the steps below to adopt the arguments used in the tool to configure Kafka in IBM Integration Bus.

  1. Set the Kerberos configuration file. It is the second argument used while running the tool on Integration server
  2. Set the producer.properties file.
    mqsichangeproperties IBNODE -c ConnectorProviders -o Kafka -n property1 -v  <full path to
    producer.properties file>

    (Optional) For Kafka consumer nodes if they are using the same properties file, please run

    mqsichangeproperties IBNODE -c ConnectorProviders -o Kafka -n property2 -v <full path to 
    producer.properties file>

  3. Configure the Kafka node with the following properties

    • Specify the topic name on the node property as <publish topic>

    • Set the 'Bootstrap servers' node property as <bootstrap server addr>

    • Set node property 'Security protocol' as <security protocol>

  4. (Optional) If you have used -Djavax.net.ssl.trustStore or -Djavax.net.ssl.keyStore while running the tool, then you need to configure the same keystore and truststore in IBM integration Bus using the following commands.

Commands to configure truststore under the Integration sever.

  1. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n truststoreFile -v 
    <Truststore value used to run the program IIBKafkaPubTestV2.jar >
  2. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n truststorePass -v 
    kafkaTrust::password
  3. mqsisetdbparms IBNODE  -n kafkaTrust::password -u temp -p <truststore password used to run
    the program IIBKafkaPubTestV2.jar >

Commands to configure keystore under the Integration server.

  1. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n keystoreFile -v <keystore
    value used to run the program IIBKafkaPubTestV2.jar >
  2. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n keystorePass  -v 
    kafkaKeystore::password
  3. mqsisetdbparms IBNODE  -n kafkaKeystore::password -u temp -p <keystore  password used to 
    run the program IIBKafkaPubTestV2.jar >

Instructions for configuring Kafka with Kerberos authentication in ACE V11.

You will need App Connect Enterprise version 11.0.0.5 or higher for Kafka nodes to use Kerberos authentication.

  1. Define the Kerberos configuration file

    a) For Integration node, run the command

    mqsichangeproperties NODE -e SERVER -o ComIbmJVMManager -n kerberosConfigFile -v <full path to 
    krb5.conf file>

  2. Set the producer.properties file using environment variable

    MQSI_KAFKA_PRODUCER_PROPERTIES_FILE and MQSI_KAFKA_CONSUMER_PROPERTIES_FILE

    export MQSI_KAFKA_PRODUCER_PROPERTIES_FILE=<full path to producer.properties file>

    b) (optional) setting the consumer.properties file for KafkaConsumer node

    export MQSI_KAFKA_CONSUMER_PROPERTIES_FILE=<full path to consumer.properties file>

  3. Configure the Kafka node with following properties

    • Specify the topic name on the node property as <publish topic>
    • Set the 'Bootstrap servers' node property as <bootstrap server addr>
    • Set node property 'Security protocol' as <security protocol>

Conclusion

The above procedure is to configure Kerberos authentication with Kafka using the producer.properties file. For those who are using plain username and password authentication instead of Kerberos authentication, you can use the same stand-alone program for testing in the following way;

  1. Change the sasl.jaas.config in the producer.properties file in the following form;
    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password";
  2. Run the program. You can use a dummy file for the argument <full path to krb5.conf file>
  3. Once you have a successful result, configure the username and password credentials as per the following Knowledge Center page for Kafka nodes to pick them.
    https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/bz91050_.htm

The stand alone program is built using Java KafkaProducer class. You can refer to the Javadoc for further information;
https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

Acknowledgement and thanks to Amar Shah for his technical support and advice in the construction of this article.


#AppConnectEnterprise(ACE)
#ACEV11
#IIBV10
#kafka
#Kerberos