DataPower

DataPower

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

DataPower UTF-16

  • 1.  DataPower UTF-16

    Posted 02/14/22 08:50 AM
    Hi All,

     I have a requirement to pass the UTF-16 incoming xml to the downstream web service. The issue is the MPG is converting the UTF-16 to UTF-8 which is truncating the emojis characters. Wondering how do we preserve the encoding. 

    Please advise.

    Regards,

    Vijay..

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------


  • 2.  RE: DataPower UTF-16

    Posted 02/14/22 09:23 AM
    Most likely the easiest way is to treat the incoming xml as non-xml. The payload should be passed to the backend unchanged.

    ------------------------------
    Charlie Sumner
    ------------------------------



  • 3.  RE: DataPower UTF-16

    Posted 02/14/22 09:36 AM
    I will need to do message translation of the incoming message. Please let me know whether i could handle this without accepting it as non-xml

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 4.  RE: DataPower UTF-16

    Posted 02/14/22 01:40 PM

    How are you setting UTF-16 encoding? Is it set on Content-Type header and/or <xml encoding>?






  • 5.  RE: DataPower UTF-16

    Posted 02/14/22 01:45 PM
    Currently it is set in the <xml encoding>. We tried to set in the content-type header as well and it did not work.

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 6.  RE: DataPower UTF-16

    Posted 02/14/22 05:38 PM
    You can't really 'preserve' the encoding per se.  DataPower parses to UTF-8 as part of it's parsing process.  You can output to UTF-16 using this statement in your XSLT:

    <xsl:output method="xml" encoding="UTF-16" />

    Charlie

    ------------------------------
    Charlie Sumner
    ------------------------------



  • 7.  RE: DataPower UTF-16

    Posted 02/14/22 07:18 PM

    I'm not sure if this is the same flow, but according to this thread, UTF-16 worked

     

    https://www.ibm.com/mysupport/s/question/0D50z00006AB7hGCAT/datapower-handling-an-encodingutf16-xml-file?language=en_US






  • 8.  RE: DataPower UTF-16

    Posted 02/15/22 10:16 AM
    Yes. I have seen this post. I did not find what Hermann suggested to resolve the issue. Can you please send that over to me in this post if you find it.

    Thank you

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 9.  RE: DataPower UTF-16

    Posted 02/15/22 12:28 PM

    I checked with Hermann. He recommends the same as Charlie's.






  • 10.  RE: DataPower UTF-16

    Posted 02/15/22 10:14 AM
    We tried applying the output format. But had no luck as well.
    <xsl:output method="xml" encoding="UTF-16" />




    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 11.  RE: DataPower UTF-16

    Posted 02/15/22 10:21 AM
    Please make sure that you are using the correct context in the results action. The context should be the output of your transform where you specified the xsl:output statement. I tried it yesterday with a simple stylesheet and it definitely worked for me. 

    If you feel that you have done everything correctly, then please open a case with IBM Support so someone can help you directly.


    ------------------------------
    Charlie Sumner
    ------------------------------



  • 12.  RE: DataPower UTF-16

    Posted 02/24/22 08:09 AM
    <xsl:output method="xml" encoding="UTF-16" />​


    This will output UTF-16 encoded XML, but ONLY if the output context of the xform action is OUTPUT, no Results action.



    ------------------------------
    Hermann Stamm-Wilbrandt
    Compiler Level 3 support & Fixpack team lead
    IBM DataPower Gateways (⬚ᵈᵃᵗᵃ / ⣏⠆⡮⡆⢹⠁⡮⡆⡯⠂⢎⠆⡧⡇⣟⡃⡿⡃)
    https://stamm-wilbrandt.de/en/blog/
    ------------------------------



  • 13.  RE: DataPower UTF-16

    Posted 02/24/22 10:26 AM
    We are actually getting UTF-8 encoded xml with UTF-16 characters. Our clients cannot change it so we are taking the incoming stream as non-xml and re-encoding with UTF-16. Thank you got it to work.

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 14.  RE: DataPower UTF-16

    Posted 02/24/22 10:48 AM

    Good that you have a workaround.

    But the situation is really bad, why stating to use XML standard, when not complying?

    I would just reject those messages and force client to send valid data if I had the power to do so.

    P.S:

    Many years ago we had a customer ticket that DataPower choked on google maps XML data.
    It turned out we correctly choked, because that XML came withoout XML declaration (meaning encoding has to be UTF-8), but it was not UTF-8 encoded.So we said our customer to raise a ticket at google.

    Obviously they did that (it was a big customer), and short time later google maps returned valid XML ;-)



    ------------------------------
    Hermann Stamm-Wilbrandt
    Compiler Level 3 support & Fixpack team lead
    IBM DataPower Gateways (⬚ᵈᵃᵗᵃ / ⣏⠆⡮⡆⢹⠁⡮⡆⡯⠂⢎⠆⡧⡇⣟⡃⡿⡃)
    https://stamm-wilbrandt.de/en/blog/
    ------------------------------



  • 15.  RE: DataPower UTF-16

    Posted 02/24/22 01:02 PM
    wow :). Yes encoding should be established part of the service contract. Sadly, we have not done it. We will talk to our vendors and see if they can send us with proper encoding style. thank you.

    ------------------------------
    Vijayasaradhi Peddysetty
    ------------------------------



  • 16.  RE: DataPower UTF-16

    Posted 12/26/22 08:33 AM
    Edited by Akshay Sawant 12/26/22 08:39 AM
    Hi 
    In a different situation we need to transform xml response to JSON before transmitting the payload further. However the xml payload is UTF-16 encoded which returns invalid xml payload response when transformed into JSON. UTF-* encoded xml get transformed correctly. We are using map policy in API manager for the transformation. Kindly help how do we overcome UTF-16 encoding issue so that the transformation to JSON is successful? 
    Thanks

    ------------------------------
    Akshay Sawant
    ------------------------------



  • 17.  RE: DataPower UTF-16

    Posted 01/05/23 09:52 AM

    Hi Akshay,
    This would  be best discussed in the API Connect community forum.

    Regards,

    Steve



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------