Co-author - Mohammed Asif Kundgol
Kafka streams data between two applications and is often used to perform real time analytics. It passes data between producers and consumers using message based topics inside a Kafka cluster. It's durability and ability to scale make it a popular choice for use in website activity tracking, collecting metrics and monitoring logs.
Using Kafka with App Connect
You can use App Connect with Kafka to perform the following operations:
- Trigger an event flow when a New message is created for a particular group ID.
- Retrieve topics based on a clause or condition.
- Send messages to a particular topic with the payload and event key ID.
To use App Connect to integrate Kafka with other connectors in the App Connect catalog, you need to connect App Connect to your Kafka environment. App Connect supports connection to the following Kafka implementations:
- Apache Kafka
- Apache Kafka on Confluent Platforms
- IBM Event Streams on premises
- IBM Event Streams on cloud
Connecting to Kafka
The following screen shot displays how you begin to connect to your Kafka account in App Connect. As displayed, the first thing you need to select is your Authorization method.
Kafka brokers can be configured with different authentication and encryption security mechanisms. Depending on how the Kafka brokers that you want to connect to have been configured, one of the following four authorization methods are required to authenticate your connection. Each method has it's own set of credentials for connecting to Kafka as displayed by the bullets.
- PLAINTEXT. Authentication via an un-authenticated and non-encrypted channel.
- Kafka brokers list: The list of Kafka brokers in the format ["x.x.x.x:9092","y.y.y.y:9092"]
2. SASL_PLAINTEXT. Authentication via Simple Authentication and Security Layer (SASL) i.e. username/password on a non-encrypted channel.
- Kafka brokers list: The list of Kafka brokers in the format ["x.x.x.x:9092","y.y.y.y:9092"]
- Username: The Kafka server username.
- Password: The Kafka server password.
- Security Mechanism: PLAIN or SCRAM-SHA-256 or SCRAM-SHA-512 security layers.
3. SASL_SSL. Authentication via Simple Authentication and Security Layer (SASL) i.e. username/password on a Secure Sockets Layer (SSL) encrypted channel.
- Kafka brokers list: The list of Kafka brokers in the format ["x.x.x.x:9092","y.y.y.y:9092"]
- Username: The Kafka server username.
- Password: The Kafka server password.
- Security Mechanism: PLAIN or SCRAM-SHA-256 or SCRAM-SHA-512 security layers.
- CA certificate: The self-signed CA certificate in PEM format used for server side authentication
4. SSL. Authentication via an SSL encryption channel.
- Kafka brokers list: The list of Kafka brokers in the format ["x.x.x.x:9092","y.y.y.y:9092"]
- CA certificate: The self-signed CA certificate in PEM format used for server side authentication.
Once you have connected Kafka you can use App Connect to create flows between Kafka and the connectors available in the App Connect catalog. The following scenarios demonstrate how you can use Kafka in App Connect to perform automation tasks.
A subscriber of a mobile service, uses App X to manage his profile. In said App X they perform such actions as updating addresses, telephone numbers, purchasing new add-on services and upgrading plans etc. All of these events from App X stream to Kafka as immutable events. These events/messages are picked up and updated in the customer's profile of a mobile customer database.
To implement this scenario in App Connect we would need to use the New message trigger in Kafka to kick off the flow. So once App X has been updated, the update is streamed to Kafka. App Connect is listening to the specified Topic xyz123 and once it detects a new message it kicks off the flow to update the mobile customer database. In this scenario the following details have been updated:
Figure 1: Kafka New message node receiving the trigger from App X
A CSV parser node from the App Connect Toolbox is used to parse the payload received from Kafka into JSON format. This is because each one of these values are required to be selected and inserted into the mobile customer database.
Figure 2: A CSV Parser node is used to parse the data received from Kafka into JSON format
As displayed in the following screenshot, a JDBC custom SQL query has been use to select the values parsed through the CSV parser to update the database with the newly updated details.
Figure 3: A JDBC custom SQL query used to insert data to the mobile customer database table using values from the parsed fields
This scenario demonstrates how you can use App Connect to automatically update a database with updated details from a mobile application.
The customer support team in Acme airlines has been tasked with tracking the latest airline information. Once this information has been received they need to contact the relevant departments with the latest flight updates in under 1 minute after the impact has been identified.
Salesforce is used to capture airline information for Acme airlines so to implement this scenario in App Connect we are going to use a custom user defined object in Salesforce called "AirInfo". This object contains fields that capture flight information such as:
A "New airinfo" Salesforce object is selected to act as the trigger so that whenever any of the above fields are updated the flow is triggered.
Figure 4: Pop up window displayed when searching for the user defined object AirInfo
A Kafka send message action has been selected as the 2nd node in the flow. It is configured to capture the following information from Salesforce in the Kafka payload:
Figure 5: Available mappings from Salesforce to be used in the Kafka payload
So when this flow runs, a message is published to Kafka containing the information configured in the payload. The customer support team can then use this message to notify the relevant teams.
This scenario demonstrates how you can use App Connect to automatically capture flight information and create a message to update relevant teams as soon as the information has been retrieved.
#AppConnect#kafka#ApacheKafka#Salesforce