webMethods

webMethods

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

Encoding issue - Converting special characters from Unix(LF) ANSI to Windows (CR LF) UTF-8

  • 1.  Encoding issue - Converting special characters from Unix(LF) ANSI to Windows (CR LF) UTF-8

    Posted Tue February 13, 2024 03:46 AM

    Hello everyone,

    I am having an issue when I try to create a record from a bizdoc. When I debug to see the values in a document by using “wm.b2b.editn:bizdocToRecord” service, some of the special characters turn into symbols. It doesn’t matter if I set the encoding parameter as UTF-8 or not.

    I also tried to use “pub.string:bytesToString” and “pub.string:stringToBytes” services with and without set encoding to get rid of those symbols but it didn’t work either.

    When I export the transaction from TN and open it in Notepad I see line endings are UNIX LF and encoding is ANSI and can see special characters without an issue as Rønne instead of R�nne.

    There are a few Java codes that I tried but the result was the same,

    byte[] ansiBytes = (byte[]) ValuesEmulator.get(pipeline, "bytes");
    StringBuilder hexString = new StringBuilder();
    for (byte b : ansiBytes) {
    String hex = Integer.toHexString(Byte.toUnsignedInt(b));
    if (hex.length() == 1) {
    hex = "0" + hex; 
    }
    hexString.append(hex);
    }
    String hexResult = hexString.toString();
    byte[] byteArray = new byte[hexResult.length() / 2];
    for (int i = 0; i < byteArray.length; i++) {
    int index = i * 2;
    int j = Integer.parseInt(hexResult.substring(index, index + 2), 16);
    byteArray[i] = (byte) j;
    }
    
    String utf8String = new String(byteArray, StandardCharsets.UTF_8);
    
    byte[] unixBytes = (byte[]) ValuesEmulator.get(pipeline, "bytes");
    
    ByteArrayOutputStream windowsOutputStream = new ByteArrayOutputStream();
    for (byte b : unixBytes) {
    if (b == '\n') {
    windowsOutputStream.write('\r');
    }
    windowsOutputStream.write(b);
    }
    byte[] windowsBytes = windowsOutputStream.toByteArray();
    
    byte[] utf8Bytes = null;
    try {
    utf8Bytes = new String(windowsBytes, "UTF-8").getBytes("UTF-8");
    } catch (UnsupportedEncodingException e) {
    e.printStackTrace();
    }
    
    String utf8String = new String(utf8Bytes, StandardCharsets.UTF_8);
    ValuesEmulator.put(pipeline,"UTF8stringOutput",utf8String); 
    

    These transactions come from a generic AS2 service so I don’t want to make changes there and asking the sender to change the content is the last option.

    Do you have any suggestions? I can provide extra info if it is needed.

    Thanks in advance.


    #webMethods