DataPower

 View Only
Expand all | Collapse all

DataPower UTF-16

  • 1.  DataPower UTF-16

    Posted Mon February 14, 2022 08:50 AM
    Hi All,

     I have a requirement to pass the UTF-16 incoming xml to the downstream web service. The issue is the MPG is converting the UTF-16 to UTF-8 which is truncating the emojis characters. Wondering how do we preserve the encoding. 

    Please advise.

    Regards,

    Vijay..

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------


  • 2.  RE: DataPower UTF-16

    Posted Mon February 14, 2022 09:23 AM
    Most likely the easiest way is to treat the incoming xml as non-xml. The payload should be passed to the backend unchanged.

    ------------------------------
    Charlie Sumner
    ------------------------------



  • 3.  RE: DataPower UTF-16

    Posted Mon February 14, 2022 09:36 AM
    I will need to do message translation of the incoming message. Please let me know whether i could handle this without accepting it as non-xml

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 4.  RE: DataPower UTF-16

    Posted Mon February 14, 2022 01:40 PM

    How are you setting UTF-16 encoding? Is it set on Content-Type header and/or <xml encoding>?






  • 5.  RE: DataPower UTF-16

    Posted Mon February 14, 2022 01:45 PM
    Currently it is set in the <xml encoding>. We tried to set in the content-type header as well and it did not work.

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 6.  RE: DataPower UTF-16

    Posted Mon February 14, 2022 05:38 PM
    You can't really 'preserve' the encoding per se.  DataPower parses to UTF-8 as part of it's parsing process.  You can output to UTF-16 using this statement in your XSLT:

    <xsl:output method="xml" encoding="UTF-16" />

    Charlie

    ------------------------------
    Charlie Sumner
    ------------------------------



  • 7.  RE: DataPower UTF-16

    Posted Mon February 14, 2022 07:18 PM

    I'm not sure if this is the same flow, but according to this thread, UTF-16 worked

     

    https://www.ibm.com/mysupport/s/question/0D50z00006AB7hGCAT/datapower-handling-an-encodingutf16-xml-file?language=en_US






  • 8.  RE: DataPower UTF-16

    Posted Tue February 15, 2022 10:16 AM
    Yes. I have seen this post. I did not find what Hermann suggested to resolve the issue. Can you please send that over to me in this post if you find it.

    Thank you

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 9.  RE: DataPower UTF-16

    Posted Tue February 15, 2022 12:28 PM

    I checked with Hermann. He recommends the same as Charlie's.






  • 10.  RE: DataPower UTF-16

    Posted Tue February 15, 2022 10:14 AM
    We tried applying the output format. But had no luck as well.
    <xsl:output method="xml" encoding="UTF-16" />




    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 11.  RE: DataPower UTF-16

    Posted Tue February 15, 2022 10:21 AM
    Please make sure that you are using the correct context in the results action. The context should be the output of your transform where you specified the xsl:output statement. I tried it yesterday with a simple stylesheet and it definitely worked for me. 

    If you feel that you have done everything correctly, then please open a case with IBM Support so someone can help you directly.


    ------------------------------
    Charlie Sumner
    ------------------------------



  • 12.  RE: DataPower UTF-16

    Posted Thu February 24, 2022 08:09 AM
    <xsl:output method="xml" encoding="UTF-16" />​


    This will output UTF-16 encoded XML, but ONLY if the output context of the xform action is OUTPUT, no Results action.



    ------------------------------
    Hermann Stamm-Wilbrandt
    Compiler Level 3 support & Fixpack team lead
    IBM DataPower Gateways (⬚ᵈᵃᵗᵃ / ⣏⠆⡮⡆⢹⠁⡮⡆⡯⠂⢎⠆⡧⡇⣟⡃⡿⡃)
    https://stamm-wilbrandt.de/en/blog/
    ------------------------------



  • 13.  RE: DataPower UTF-16

    Posted Thu February 24, 2022 10:26 AM
    We are actually getting UTF-8 encoded xml with UTF-16 characters. Our clients cannot change it so we are taking the incoming stream as non-xml and re-encoding with UTF-16. Thank you got it to work.

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 14.  RE: DataPower UTF-16

    Posted Thu February 24, 2022 10:48 AM

    Good that you have a workaround.

    But the situation is really bad, why stating to use XML standard, when not complying?

    I would just reject those messages and force client to send valid data if I had the power to do so.

    P.S:

    Many years ago we had a customer ticket that DataPower choked on google maps XML data.
    It turned out we correctly choked, because that XML came withoout XML declaration (meaning encoding has to be UTF-8), but it was not UTF-8 encoded.So we said our customer to raise a ticket at google.

    Obviously they did that (it was a big customer), and short time later google maps returned valid XML ;-)



    ------------------------------
    Hermann Stamm-Wilbrandt
    Compiler Level 3 support & Fixpack team lead
    IBM DataPower Gateways (⬚ᵈᵃᵗᵃ / ⣏⠆⡮⡆⢹⠁⡮⡆⡯⠂⢎⠆⡧⡇⣟⡃⡿⡃)
    https://stamm-wilbrandt.de/en/blog/
    ------------------------------



  • 15.  RE: DataPower UTF-16

    Posted Thu February 24, 2022 01:02 PM
    wow :). Yes encoding should be established part of the service contract. Sadly, we have not done it. We will talk to our vendors and see if they can send us with proper encoding style. thank you.

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 16.  RE: DataPower UTF-16

    Posted Mon December 26, 2022 08:33 AM
    Edited by Akshay Sawant Mon December 26, 2022 08:39 AM
    Hi 
    In a different situation we need to transform xml response to JSON before transmitting the payload further. However the xml payload is UTF-16 encoded which returns invalid xml payload response when transformed into JSON. UTF-* encoded xml get transformed correctly. We are using map policy in API manager for the transformation. Kindly help how do we overcome UTF-16 encoding issue so that the transformation to JSON is successful? 
    Thanks

    ------------------------------
    Akshay Sawant
    ------------------------------



  • 17.  RE: DataPower UTF-16

    Posted Thu January 05, 2023 09:52 AM

    Hi Akshay,
    This would  be best discussed in the API Connect community forum.

    Regards,

    Steve



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------