DataPower

Hi, I'm Krithika Prakash - Senior Technical Staff Member (STSM) in IBM APIConnect & DataPower Product Development team.

In this two part article, I have covered the following topics: 
1) In Part 1, I  have given an overview of the main Kafka features supported in DataPower and how these features can be used in a variety of use cases involving Kafka traffic. 
2) In Part 2, I have covered step by step hands on tutorial on how DataPower can be configured for a specific Kafka security use case.



Please read Part 1 before proceeding on this article. I'll now demo step by step details on how you can configure DataPower for this scenario. That should also give you an insight in everything you can do in DataPower with respect to Kafka support.

Setup Details
For the sake of this demo, I'm using two DataPower systems  - DP1 (assume this is in DC1) and DP2 (assume this is in DC2) and two different Kafka topics - krithi-test-DC1 and krithi-test-DC2. Note that we will be reading and writing JSON type messages for the sake of this demo.

Kafka Server Setup
First let's look at the Kafka server setup. I'm using a UI client (Conduktor) that lets us see what's going on in the Kafka cluster.
I created two topics - krithi-test-DC1 and krithi-test-DC2. Once you have the DataPower setup ready, you can come back to this UI interface to push messages to DC1 topic and see what  is received in DC2 topic.

Data Center 1 simulation 
- First create a Kafka Cluster object in DP1.

- Next create a MPGW to handle JSON payload


- In MPGW, configure a Kafka Front Side handler to read from krithi-test-DC1 topic. Also you can add a consumer group name here.

- In the processing policy of MPGW, create a JWE encrypt action. I created keys and certificates in this system using DataPower Crypto tools.

- For the backend URL, specify the HTTPS URL and port number where DP2 will be configured to listen.


This completes DataPower 1 or (DataCenter 1) setup

Data Center 2 simulation
- First create a Kafka Cluster object in DP2, same as in DP1.
- Next create a MPGW to handle JSON payload, same as in DP1.
- Configure a HTTPs Front Side handler with the same port that we configured in DP1 to send HTTP traffic to .

- In the processing policy of MPGW, create a JWE decrypt action. I imported the private key that corresponds to the certificate used in the other DataPower instance. 


- Let's add a small twist to this, so that we can double check that the messages really get encrypted. Let's add a match policy to do decryption only if the header "decrypt=true" is present. If not, by default only encrypted messages will be pushed to the Kafka topic.

- For the backend URL, specify the Kafka URL to push to DC2 topic : dpkafka://kafkaTest?RequestTopic=krithi-topic-dc2
This completes DataPower 2 or (DataCenter 2) setup.

Let's now test end to end.
Back to the UI client, let's send a message to krithi-topic-DC1, but before we do that let's start listening on messages on krithi-topic-DC2.


The messages sent to DC1 should be read by DataPower 1, encrypted and sent using HTTPs to DataPower 2 where it will be pushed to DC2 topic .  Sending JSON message, I can see the message shows up as encrypted in DC2. This proves the fact that messages are actually getting encrypted and secured between two DCs.



Now let me add the "decrypt:true" header to the original Kakfa request. As you can see the headers are passed as is from Kafka DC1 topic  ---> DP1 in DC1  --> DP2 in DC2 where it picked up that header and matched the header policy to perform JWE decryption before pushing to DC2.  Hence you see the decrypted message here.



This concludes the demo. I covered how DataPower can be used as a Kafka client supporting both consumer and publisher scenarios and how Kafka messages can be encrypted/decrypted within DataPower to secure traffic between data centers. 

Attached are the DataPower MPGW configuration exports used in this tutorial: DataPowerKafkaSupport

Hope you find this article series useful and let us know if you have any feedback or comments. Thank you !

PS: I would like to thank my colleague Sangeetha Ilango for helping me with the setup to simulate this scenario in the lab and also my colleague Francisco Moraes for sharing his expertise and knowledge in this area of Kafka support in DataPower.