MQ

MQ

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

Codepage support in MQ REST api

  • 1.  Codepage support in MQ REST api

    Posted Mon August 24, 2020 11:40 AM

    I'm trying to send a JSON message with non UTF-8 content, and I am unable to specify any other CHARSET via the content-type header but UTF-8. for any other CHARSET the following error is returned:

    "MQWB0107E: Unable to parse the request data due to exception"

    Does the MQ REST api support any non UTF-8 character set ?



    ------------------------------
    Dan Kalmar
    ------------------------------


  • 2.  RE: Codepage support in MQ REST api

    Posted Mon August 24, 2020 11:51 AM
    Hi Dan,

    It only supports UTF-8 as documented here: https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.2.0/com.ibm.mq.ref.dev.doc/q130750_.htm

    Support for binary formats, and other character sets is on the list of things we would like to do at some point.

    Regards, Matt.

    Matt Leming
    Architect, IBM MQ for z/OS

    Email: lemingma@uk.ibm.com
     
     





  • 3.  RE: Codepage support in MQ REST api

    Posted Mon August 24, 2020 03:09 PM
    Hopefully the REST API will support other charsets so it can become usefull for non english-like languages.

    ------------------------------
    Dan Kalmar
    ------------------------------



  • 4.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 04:38 AM

    Hi Dan. Your non english characters should work with UTF-8.

    UTF-8 is not a charset. It is a character encoding system that works well with Arabic, Russian and other non English language characters. Each unicode character can be encoded in UTF-8.

    Still it were nice if IBM also supported UTF-16 as encoding system.



    ------------------------------
    Matthias Jungbauer
    ------------------------------



  • 5.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 05:40 AM
    It is interesting what you are saying.
    I have hebrew characters in some JSON fields inside the message. These hebrew characters all end up on the MQ inbound queue as ASCII value x'3F' (QUESTION MARK), so some incorrect translation is taking place.
    On MQ explorer I can see the character set identifier associated with the message on the queue is 1208. 
    Can you point me to a document showing that hebrew is supported in UTF-8 ?

    ------------------------------
    Dan Kalmar
    ------------------------------



  • 6.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 05:55 AM

    I just googled "Hebrew characters in UTF-8" and the first hit was this:

    https://www.utf8-chartable.de/unicode-utf8-table.pl?start=1280

    search for the word "hebrew" on this page.

    Can you say a little more about how you are providing your JSON message to the REST interface? Is it a file? Is the file in UTF-8?

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    ------------------------------



  • 7.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 06:45 AM
    I can see the hebrew characters appear in the table as double byte codes. Not sure I understand how those would be represented in the MQ queue where MQ Explorer shows the message data in single byte charcters.
    I am sending the JSON message from a Windows REST client called INSOMNIA (https://insomnia.rest/)


    ------------------------------
    Dan Kalmar
    ------------------------------



  • 8.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 09:07 AM
    The last few slides of this MQTC session on Data Conversion talk about Unicode (which maps writing symbols to a decimal value) and the Unicode Transformation Formats like UTF-8 and UTF-16 (which maps Unicode decimal values to a computer based binary encoding scheme) -> https://mqtechconference.com/sessions_v2018/MQTC_v2018_DataConversion.pdf

    Pretty much any written language in the world is included in Unicode.  The Unicode consortium did reject Klingon, though. :-)

    ------------------------------
    Tim Zielke
    ------------------------------



  • 9.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 12:07 PM
    OK. I saved the JSON message with UTF8 encoding into a file in a PC folder.
    Then viewed the hex contents using "Hex editor Neo". 
    The hebrew characters are correctly stored as 2 byte codes according to the UTF-8 encoding for hebrew.
    Then I sent this file via CURL to the z/OS MQ inbound queue using the RESP API.
    The hebrew characters were converted to x'3F' single byte ASCII values. (question-mark) 
    So, what am I doing wrong ?


    ------------------------------
    Dan Kalmar
    ------------------------------



  • 10.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 12:29 PM

    You told curl that it is utf-8?
    When you use MQ Explorer. Does the message look correct in hexadecimal format?



    ------------------------------
    Matthias Jungbauer
    ------------------------------



  • 11.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 04:05 PM
    Yes, I told curl by specifying:    -H "Content-Type: application/json;charset=utf-8" 
    and when I view the message data in MQ explorer the hebrew characters appear as x'3F' still.
    The character set identifier of the message as shown by MQ explorer is 1208 = UTF8.

    ------------------------------
    Dan Kalmar
    ------------------------------



  • 12.  RE: Codepage support in MQ REST api

    Posted Tue August 25, 2020 06:18 PM
    Does the hex view of the message look correct?
    Is this just Explorer not being able o display the characters in text view?

    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    ------------------------------



  • 13.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 01:58 AM
    Yes, in the hex view each hebrew character is replaced by a single hex value of '3F'.

    ------------------------------
    Dan Kalmar
    ------------------------------



  • 14.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 05:18 AM
    Hi Dan. Posting a message with Hebrew chars into the cloud qmgr works.
    -H "Content-Type: text/plain;charset=utf-8" --data @hebrew.utf8
    I will check if MQ Explorer displays it correctly.


    ------------------------------
    Matthias Jungbauer
    ------------------------------



  • 15.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 05:50 AM
    In field message data MQ explorer displays it correctly.
    The byte stream in message data byte looks correct, too.
    Just the right hand side looks literally questionable.


    ------------------------------
    Matthias Jungbauer
    ------------------------------



  • 16.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 06:40 AM
    In my case, I send to a z/OS queue manager and I send JSON and not plain text. So there could be something different. 
    Also my MQ software level on the z/OS side is 2 years old, so maybe this issue could be resolved by upgrading to the latest level although I did not see any fix that addresses an issue with UTF8 encoding.

    ------------------------------
    Dan Kalmar
    ------------------------------



  • 17.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 07:07 AM
    -H "Content-Type: application/json;charset=utf-8" --data @hebrew.json

    No idea. The long list of fixes indicates a long list of bugs. Some of which are still unknown.
    Are any other tools in your environment having issues when exchanging data between z/os and windows. E.g. when you send a utf8 file from Windows to z/os and back. Is it still utf8 or something else.
    Or when you put a utf8 message in z/os into the queue. Is MQ explorer then able to display it correctly?


    ------------------------------
    Matthias Jungbauer
    ------------------------------



  • 18.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 08:32 AM
    Hi,

    I had a quick try of this against my development queue manager which is "kind of" at MQ 9.2 for z/OS.

    I can put JSON payload with Hebrew content on it via the REST API, and then get it via the REST API and the contents of the message comes back correctly.

    Regards, Matt.

    Matt Leming
    Architect, IBM MQ for z/OS

    Email: lemingma@uk.ibm.com
     
     





  • 19.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 10:51 AM
    Edited by Dan Kalmar Wed August 26, 2020 10:51 AM
    Matt
    When you put a JSON message with hebrew characters on the z/OS queue, can you try to view the hex data via MQ Explorer and see the hebrew double byte hex values ?
    Cheers....Dani

    ------------------------------
    Dan Kalmar
    ------------------------------



  • 20.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 11:31 AM
    Hi Dan,

    Here is what I get with the 9.1.5 version of MQ REST API. The data is ending up on the queue OK, and the message data in Explorer is OK. The double byte values are correct, but threw me for a bit, but they aren't displayed correctly in the message data bytes field. I get the same result if I switch to the 9.0.5 REST API code.





    Regards, Matt.

    Matt Leming
    Architect, IBM MQ for z/OS

    Email: lemingma@uk.ibm.com
     
     





  • 21.  RE: Codepage support in MQ REST api

    Posted Wed August 26, 2020 02:05 AM
    Edited by Dan Kalmar Thu August 27, 2020 04:10 PM

    The CURL trace shows the data is sent up with correct hex values.
    The MQWEB server log however has the message data already converted with x'3F' instead the double byte
    hebrew characters.
    Still a mystery where the data gets altered on the way.

     



  • 22.  RE: Codepage support in MQ REST api

    Posted Tue September 19, 2023 12:56 AM

    Was this ever solved?

    I have used the sample dotnet vb SimplePut example to write to a queue; and then using MQ 9.3.3 REST API for messaging; and it fails as

    {
        "error": [
            {
                "msgId": "MQWB0309E",
                "action": "Resubmit the request, correcting any issues.",
                "completionCode": 0,
                "reasonCode": 0,
                "type": "rest",
                "message": "MQWB0309E: The received or browsed message type is unsupported by the mqweb server.",
                "explanation": "The mqweb server successfully received an MQ message, but was of an unsupported message type."
            }
        ]
    }

    and when using `amqsbcgc` the output is

    AMQSBCG0 - starts here
    **********************
    
     MQOPEN - 'TEST.QUEUE.1'
    
    
     MQGET of message number 1, CompCode:0 Reason:0
    ****Message descriptor****
    
      StrucId  : 'MD  '  Version : 2
      Report   : 0  MsgType : 8
      Expiry   : -1  Feedback : 0
      Encoding : 546  CodedCharSetId : 1200
      Format : '        '
      Priority : 0  Persistence : 0
      MsgId : X'414D5120415050543032202020202020C55AC26401C47AA7'
      CorrelId : X'000000000000000000000000000000000000000000000000'
      BackoutCount : 0
      ReplyToQ       : '                                                '
      ReplyToQMgr    : 'QM1                                          '
      ** Identity Context
      UserIdentifier : 'user1     '
      AccountingToken :
       X'160105150000007E46FCF3D997C6E2B1A0FF3F97DD000000000000000000000B'
      ApplIdentityData : '                                '
      ** Origin Context
      PutApplType    : '11'
      PutApplName    : 'ePut\bin\Debug\SimplePut.exe'
      PutDate  : '20230919'    PutTime  : '04395225'
      ApplOriginData : '    '
    
      GroupId : X'000000000000000000000000000000000000000000000000'
      MsgSeqNumber   : '1'
      Offset         : '0'
      MsgFlags       : '0'
      OriginalLength : '-1'
    
    ****   Message      ****
    
     length - 24 of 24 bytes
    
    00000000:  7400 6500 7300 7400 2000 6D00 6500 7300           't.e.s.t. .m.e.s.'
    00000010:  7300 6100 6700 6500                               's.a.g.e.        '
    
    
    
     No more messages
     MQCLOSE
     MQDISC


    ------------------------------
    om prakash
    ------------------------------



  • 23.  RE: Codepage support in MQ REST api

    Posted Tue September 19, 2023 01:19 AM

    Hi Om,

    You appear to be suffering from a different problem. This thread is about string messages in different codepages. Your message is a binary message not a string message.

    I expect when you put the message with the SimplePut.exe executable you probably intended it to be a string message, but its format is blank (which means binary) and the REST API only supports string messages. Perhaps you can correct the SimplePut.exe executable to fill in the MQMD Format field correctly, or use another application to put your test message that already puts MQSTR as the value in the MQMD Format field. amqsput is one such example application.

    Then try retrieving that message with the REST API.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------



  • 24.  RE: Codepage support in MQ REST api

    Posted Wed September 20, 2023 10:00 AM

    thanks Morgan. I missed seeing the Format was blank; and also the utility sample IBM provides does not add the format to be MQSTR.



    ------------------------------
    om prakash
    ------------------------------