DataPower

 View Only
  • 1.  Datapower V10 Kafka FSH

    Posted Thu March 10, 2022 11:09 AM
    Hi,

    I am trying to read the message from Kafka using Url-open.

    https://www.ibm.com/docs/en/datapower-gateway/10.0.1?topic=open-url-kafka

    For example below is the sample JSON data from Kafka,

    {

    "source_id" : "xxxxxx",

    "source" : "IRD-US",

    "action_type" : "U",

    "country_code" : "US",

    "party_type" : "ACCOUNT",

    "party_level" : "TX",

    "party_status" : "ACTIVE",

    "changed_fields" : [ "relatedAccounts.ciAccounts" ]

    }


    I need to read the messages which is having country code is equal to US only("country_code" : "US"). Is it possible.

    Thanks,
    Aamira.B

    ------------------------------
    Aamira Badsha
    ------------------------------


  • 2.  RE: Datapower V10 Kafka FSH

    Posted Wed March 23, 2022 01:25 PM
    Hi Aamira,

    You would be better suited in GatewayScript (JavaScript) since JSON support is native there.  You can use the urlOpen https://www.ibm.com/docs/en/datapower-gateway/10.0.x?topic=apis-urlopen-module then after you would want to read as json: https://www.ibm.com/docs/en/datapower-gateway/10.0.x?topic=apis-urlopen-module#response.readasjson

    At that point you can parse the json data which have the country_code US or not, skip the ones that do not.

    ------------------------------
    DOMINIC MICALE
    ------------------------------



  • 3.  RE: Datapower V10 Kafka FSH

    Posted Fri March 25, 2022 09:47 AM

    Hi Aamira and Dominic,

    Given the link Aamira provided to our documentation, I'm assuming to connect to Kafka Aamira is trying a target url of dpkafka://cluster?queryParameters which is only supported by the xslt dp:url-open extension element.  Dom is correct that handling JSON data is best suited for GatewayScript, but unfortunately the GatewayScript urlopen.open function only supports the http, https, mq and dpmq protocols.  Thus to get the data you will need to use xslt, but the data returned is not XML, thus the response attribute of the dp:url-open function must be either binaryNode or  responsecode-binary, but since binaryNode is deprecated, you should use responsecode-binary which will return an XML document like this:

    <result>
      <binary>***BINARY NODE***</binary>
      <responsecode>200</responsecode>
      <reasonphrase>OK</reasonphrase>
      <httpversion>...</httpversion>
      <headers> ... </headers>
    </result>

    You can then get the binary node data in to a string by using the dp:binaryNodeToString extension function (see https://www.ibm.com/docs/en/datapower-gateway/10.0.1?topic=functions-dpbinarynodetostring) , something like

    <xsl:variable name="kafkaresponse" select="dp:binaryNodeToString($urlopenresponse/result/binary)" />

    As for then processing the JSON response, I would suggest using dp:gatewayscript (see https://www.ibm.com/docs/en/datapower-gateway/10.0.1?topic=functions-dpgatewayscript) where you can pass this json string to a .js file, parse this string into a JSON object and search the data using standard JavaScript programming of handling JSON data.

    Regards,
    Steve



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 4.  RE: Datapower V10 Kafka FSH

    Posted Fri March 25, 2022 12:08 PM

    Thanks Dominic for pointing out that this is actually possible with GatewayScript on its own.  I've used my previous post in the past as a workaround for retrieving data using protocols that were not supported by GatewayScript, but our documentation team has been made aware that the GatewayScript urlopen module section needs to include the dpkafka protocol in addition to HTTP and MQ.

    Here's  an example I was provided:

    var urlopen = require('urlopen');
    var hm = require('header-metadata');
    var open_options = {
      target: 'dpkafka://kafka-test;RequestTopic=request&Key=123456&KeyType=binary',
      data: {withdraw: 100},
      timeout: 10
    };
    urlopen.open(open_options, function (err, response) {
      if {error} {
        session.output.write({ error: "openCallback error: " + err.errorMessage });
      } else {
        response.readAsBuffer(function(readError, data) {
          if (readError) {
            session.output.write({error: "ReadAsBuffer Error: " + readError.toString() }};
          } else {
            // output the data or do something meaningful with it.
            session.output.write(data);
          }
        });
      }
    });

     I don't have a lot of experience with this protocol, but  I 'm assuming that kafka-test is a Kafka Cluster DataPower object name that provides the connectivity to your Kafka environment.

    Regards,
    Steve



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------