DataPower

DataPower

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

Charset=UTF-8 error on Datapower

  • 1.  Charset=UTF-8 error on Datapower

    Posted Thu October 29, 2020 08:51 AM

    Dear Team,

    on Datapower 2018.4.1.9 i have created mpgw with xml to json transform policy .xsl.

    when we are sending xml request with Arabic characters it is getting failed with parser error.

    our request type is XML and response type is pass through.

    when we are sending UTF-8 encoding from SOAPui, our request is transformed properly.

    whenever we are not sending that from soapui , request is failed with parses error. (encoding UTF-8 in soapui Properties)

    Please suggest what to do.

    i have doubt this bug is related to Datapower 2018.4.1.9

    can you suggest any work around for this.

    Kind Regards,

    Sanket.



    #DataPower
    #Support
    #SupportMigration


  • 2.  RE: Charset=UTF-8 error on Datapower

    Posted Thu October 29, 2020 09:54 AM

    Hi San,

    what is the DataPower error you are receiving when there isn't an XML declaration. And by the way, are the other two threads mentioning Arabic characters related to the same issue?

    --HP



    #DataPower
    #Support
    #SupportMigration


  • 3.  RE: Charset=UTF-8 error on Datapower

    Posted Thu October 29, 2020 11:00 AM

    Dear Hermann,

    Below error:

    11:55:14 AMmpgwerror593346error10.1.79.900x00030001mpgw (fff): Parse error11:55:14 AMmultisteperror593346request10.1.79.900x80c00008mpgw (fff): rule (fef_rule_0): Implied action Parsing input as XML, attempt pipeline. failed: illegal character 0xc7 at offset 115 of http://10.1.79.90:8095/11:55:14 AMxmlparseerror593346request10.1.79.90 0x80e003aampgw (fff): illegal character 0xc7 at offset 115 of http://0.0.0.0:8095/

    yes same issue but today i found this bcoz of UTF-8 encoding my request is failing.

    Can you please suggest work around for the same.



    #DataPower
    #Support
    #SupportMigration


  • 4.  RE: Charset=UTF-8 error on Datapower

    Posted Thu October 29, 2020 01:38 PM

    I have had the same problem a couple of times with Scandinavian characters and in my case it has never been a DataPower problem. Usually it helps to verify the issue with other client such as curl or Postman.


    And am I guessing correctly that when you receive the message from IIB it comes through MQ? When integrating with MQ you'll have to remember that MQ has "internal encoding" (CCSID) that for example in our case is always 819 (Latin 1). So if receiving messages through MQ always remember to check that the CCSID values are the ones that you are expecting.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 5.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 04:47 AM

    Dear Hermann,

    yes ccsid is 819 ,

    when am using response.readAsXML it is getting failed ata first arabic character,

    if am using response.readAsBuffer it will give us response but not able to parse that response .


    if we add xml declare in the response using concat , our data is parsed but our arabic characters are changed to ????? like this.


    please suggest how to keep the data same after buffer response and parse without arabic character changing.



    #DataPower
    #Support
    #SupportMigration


  • 6.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 05:45 AM

    There is a solution to a problem on mqseries forum that matches your description:

    http://www.mqseries.net/phpBB/viewtopic.php?p=214984&sid=fee8389a4491e3444e42f9504e066053

    See if you are able to use the solution provided in the thread above.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 7.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 07:17 AM

    not working for me, because m calling mq from gatewayscript.




    #DataPower
    #Support
    #SupportMigration


  • 8.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 08:41 AM

    Are you sure? Although the thread states that the CCSID config is defined in handler, it is actually defined in queue manager object.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 9.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 08:41 AM

    Are you sure? Although the thread states that the CCSID config is defined in handler, it is actually defined in queue manager object.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 10.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 09:16 AM

    yes herman , when i assign 720 in CCSID according to the link , my Queue manager object goes down, so am using 1208 in CCSID.



    #DataPower
    #Support
    #SupportMigration


  • 11.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 01:06 PM

    Ok. 1208 should work. Or at least seems to work for me. When I get a message from queue using url-open and parse it using readasxlm() I can see that Arabic characters are transmitted ok.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 12.  RE: Charset=UTF-8 error on Datapower

    Posted Fri October 30, 2020 01:22 PM

    Dear Hermann,


    Can you check my below code.

    when i use readasXML i am getting parse error, but when i use readAsBuffer i am getting response with arabic char.


    It is failing at parse when i use readAsXML.


    var options =

    {

    target: 'dpmq://RB/?RequestQueue=requestQueue;ReplyQueue=replyQueue;timeout=6000',

    data : Input,

    headers : { MQMD : { // JSON object for specified header_name

    MQMD: { // JSON object for the MQMD header

    StructId : { $ : 'MD' } ,

    Version : { $ : '1'} ,

    }

    }

    }

    };

    urlopen.open (options, function(error, response) {

    if (error) {

    // an error occurred during the request sending or response header parsing

    //session.output.write("urlopen error: "+JSON.stringify(error));

    } else {

    // get the response status code

    var responseStatusCode = response.statusCode;

    console.log("Response status code: " + responseStatusCode);

    response.readAsBuffer(function(error, responseData){

    if (error){

    throw error ;

    } else {

    console.error(response.get({ type: 'mq'}, 'MQMD'));

    console.error(responseStatusCode);

    console.error(options);

    console.error(responseData);

    //apim.output('text/xml');

    session.output.write(responseData) ;

    }

    });

    }

    });



    #DataPower
    #Support
    #SupportMigration


  • 13.  RE: Charset=UTF-8 error on Datapower

    Posted Mon November 02, 2020 05:52 AM

    I think it's not a code issue. Just make sure that your queue manager definition has "Convert Input" on and CCSID is 1208. It the response is still failing, I'd check that you are getting back valid XML.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 14.  RE: Charset=UTF-8 error on Datapower

    Posted Mon November 02, 2020 08:15 AM

    Dear Herman,,


    Am getting xml response with arabic characters.

    But while parsing am getting error Parser Error.

    xml which is received from MQ is ISO-8859-1

    MQ is not giving xml declaration in response.


    how can we parse this xml with arabic data encoding ISO-8859-1


    Please suggest.



    #DataPower
    #Support
    #SupportMigration


  • 15.  RE: Charset=UTF-8 error on Datapower

    Posted Mon November 02, 2020 08:53 AM

    ISO-8859-1 doesn't support arabic chars, please use UTF-8 (CCSID 1208) as mentioned in my previous post.


    --HP



    #DataPower
    #Support
    #SupportMigration


  • 16.  RE: Charset=UTF-8 error on Datapower

    Posted Mon November 02, 2020 09:32 AM

    HI Hermann


    IIB is giving ISO-8859-6 encoding with ccsid 819, we tried to append the same in response , but our arabic data is getting changed



    #DataPower
    #Support
    #SupportMigration


  • 17.  RE: Charset=UTF-8 error on Datapower

    Posted Tue November 03, 2020 06:11 AM

    Again, CCSID 819 is Latin 1=ISO-8859-1, arabic chars not supported. In DataPower QM definition on CCSI tab you should set Convert = on and CCSID to 1208 or something that supports arabic characters.


    --HP



    #DataPower
    #Support
    #SupportMigration