MQ

MQ

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  MSGID to timestamp conversion

    Posted Mon December 21, 2020 08:00 PM

    HI

    It is possible to 'reverse engineer' an MQ MSGID to extract the timestamp? Does anyone have a formula to share?

    Some sample data...

    Dec 21, 2020 4:11:09 PM 414D5120 475731202020202020202020 B393D9 5F23 2906 08
    Dec 21, 2020 4:11:11 PM 414D5120 475731202020202020202020 B393D9 5F23 2906 0A
    Dec 21, 2020 4:11:12 PM 414D5120 475731202020202020202020 B393D9 5F23 2906 0C
    Dec 21, 2020 4:11:12 PM 414D5120 475731202020202020202020 B393D9 5F23 2906 0E

    The date/times above are PST.
    The first 16 bytes are AMQ + space + QueueManager name in ASCII. It would appear that the last byte is just a sequence number.)

    Thanks  



    ------------------------------
    hirschel wasserman
    ------------------------------


  • 2.  RE: MSGID to timestamp conversion

    Posted Tue December 22, 2020 01:04 PM
    Edited by Tim Zielke Tue December 22, 2020 04:43 PM

    Have you asked IBM support if they will provide the algorithm?  Based on my research/educated guess for Linux systems running MQ 9.1, the msg-id portion that holds a "timestamp" would be something like this using one of your MsgIds "B393D9 5F23 2906 08":

    B393D95F (was anticipating it in the big endian byte order of 5FD993B3 based on the endianness of 23290608) is the Unix epoch time of 5FD993B3 = 12/16/2020 at 4:57:23 AM GMT.

    23290608 = 232906 (a value that seems to increment by 1 roughly every 2 seconds from the previous epoch value) + the last byte 06 which is some kind of sequence number that gets incremented per message created during the 232906 interval.

    I couldn't figure out the 232906 portion well enough to have some formula to reverse engineer back to a specific timestamp.  My guess is that you might not be able to reverse engineer it back to a timestamp, if the 232906 portion is not incremented in a consistent time interval, which is what it looks like could be the case to me.  

    If IBM does provide the formula and you can share it, please do!

    <Edit>

    One other thought.  Maybe the 23290608 piece represents a counter of unique message ids since the epoch value of 5FD993B3 = 12/16/2020 at 4:57:23 AM GMT.  Just an educated guess, though.



    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 3.  RE: MSGID to timestamp conversion

    Posted Wed December 23, 2020 04:50 PM

    HI Tim ... thanks for putting some thought into this.

    I checked with the MQ Support folks, According to them it is not possible to reverse engineer the timestamp.

    "The date and time is included in the overall assignation, but also an internal counter which always increases and which is used by other objectIDs"

    C'est la vie.


    Hirschel Wasserman

     

    This email and any attachments are intended only for the named recipient and may contain confidential and/or privileged material. Any unauthorized copying, dissemination or other use by a person other than the named recipient of this communication is prohibited. If you received this in error or are not named as a recipient, please notify the sender and destroy all copies of this email immediately.

    Insurance Corporation of British Columbia | 151 W. Esplanade | North Vancouver | V7M 3H9
    Contact Us




    Original Message


  • 4.  RE: MSGID to timestamp conversion

    Posted Fri December 25, 2020 04:03 PM

    Hi Hirschel,

    Thanks for sharing that info! So if I had to guess, those 8 bytes at the end that make the message id unique sound like they are the following on Unix systems:

    4 bytes (unix epoch time of some point in the recent past) + 4 bytes (counter of unique message ids/other ids since the epoch time in the previous 4 bytes)

    It looks like both are 4 byte binary integers, so endianness would matter. For example, x86 would store an epoch time of 5FD993B3 as B393D95F (little endian) and AIX would store 5FD993B3 as 5FD993B3 (big endian). Mentioning this in case it helps someone else trying to decipher the data.

    Thanks,

    Tim



    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 5.  RE: MSGID to timestamp conversion

    Posted Sat December 26, 2020 10:52 PM

    I did a little experimenting.

    On Z/OS MQ, there appeared a correlation to a epoch-like date/time for bytes 1-4.  The number of datetime seconds appeared matched the epoch-seconds.

    PST Datetime                                     header                                                                        Byte17-20            bytes21-24          HEX2BIN

    Dec 26 2020 5:38:01 PM               C3E2D840D4D8D4F14040404040404040                D90787DF            BF1AA542            3,641,149,407

    Dec 26 2020 5:38:06 PM               C3E2D840D4D8D4F14040404040404040                D90787E4             A4AD6841           3,641,149,412

    Dec 26 2020 5:38:09 PM               C3E2D840D4D8D4F14040404040404040                D90787E7             AE9EE742             3,641,149,415

    Dec 26 2020 5:38:11 PM               C3E2D840D4D8D4F14040404040404040                D90787E9             71FE2D42             3,641,149,417

    Dec 26 2020 5:38:16 PM               C3E2D840D4D8D4F14040404040404040                D90787EE             07E3E542             3,641,149,422

     

    But this was not consistent over a longer period of time.

     

    2020-12-26 18:28:38                       D9079330                             3,641,152,304

    2020-12-26 19:19:28                       D9079E8A                            3,641,155,210

    3050                                                                                    2,906                    Difference in seconds

     

    Comparing different Z/OS Queue Managers, similar MSGID format in bytes 17-24.

     

    ON Linux, the data was quite different, and the increment was in the 21th byte, for this sample.  The hex increment does not correspond with the datetime increment.

    Dec 26 2020 6:49:14 PM

    414D5120514D47575342582020202020

    1D4F875F

    03C56C23

    Dec 26 2020 6:50:04 PM

    414D5120514D47575342582020202020

    1D4F875F

    06C56C23

    Dec 26 2020 6:52:17 PM

    414D5120514D47575342582020202020

    1D4F875F

    09C56C23

    Dec 26 2020 6:52:20 PM

    414D5120514D47575342582020202020

    1D4F875F

    0BC56C23

    Dec 26 2020 6:52:25 PM

    414D5120514D47575342582020202020

    1D4F875F

    0DC56C23

    Comparing two Linux Queue Managers, very different MSGID in bytes 17-24.

     

    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    Hirschel Wasserman

     

    This email and any attachments are intended only for the named recipient and may contain confidential and/or privileged material. Any unauthorized copying, dissemination or other use by a person other than the named recipient of this communication is prohibited. If you received this in error or are not named as a recipient, please notify the sender and destroy all copies of this email immediately.

    Insurance Corporation of British Columbia | 151 W. Esplanade | North Vancouver | V7M 3H9
    Contact Us




    Original Message


  • 6.  RE: MSGID to timestamp conversion

    Posted Mon December 28, 2020 09:42 AM
    Edited by Tim Zielke Mon December 28, 2020 09:51 AM

    z/OS (mainframe) is an entity unto itself when compared to distributed computers like Linux, Windows, etc., so I would not anticipate it using Unix epoch time for its unique MQ message id algorithm.  I at least don't see anything that looks like Unix epoch time to me in those z/OS message ids that you provided.

    For the Linux message ids you pasted, I assume they were generated on an x86 processor which stores 4 byte integers in a little-endian or "reversed" format.  Therefore, I would read the last 8 bytes in the first message id as follows:

    Dec 26 2020 6:49:14 PM 414D5120514D47575342582020202020 1D4F875F 03C56C23

    5F874F1D (big endian representation of 1D4F875F) = 10/14/2020 at 7:18:53 pm GMT in Unix epoch time

    236CC503 (big endian representation of 03C56C23) = 594,330,883 which looks like some counter that has been increased that amount since the Unix epoch time of 10/14/2020 at 7:18:53 pm GMT in Unix epoch time

    When you put the last 4 bytes of your 5 Linux messages in big endian, you can see they are marginally incremented in the last byte:

    236CC503

    236CC506

    236CC509

    236CC50B

    236CC50D



    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message


  • 7.  RE: MSGID to timestamp conversion

    Posted Tue December 29, 2020 05:26 AM
    It's a VERY long time since I was familiar with the Victory Qmgr code but I know that the generated MsgId was based on the STCK value.




    Original Message


  • 8.  RE: MSGID to timestamp conversion

    Posted Tue December 29, 2020 10:11 AM

    Thanks, Dermot! That would make sense and be something specific to z/OS. For those not aware, STCK is a mainframe assembler instruction (Store Clock) to get a time of day clock value.

    It might be worth noting on the distributed side that the Unix epoch time ends in 2038. If you let it wrap, you probably still have the range of 1/1/1970 until the time in the 1990s when IBM MQ became in use on distributed to have unique Unix epoch time values for the product lifetime of IBM MQ distributed. However, at some point in the 2060s you will have duplicate values for Unix epoch time for message ids. The range of the 4 byte Unix epoch time is about 68 years.



    ------------------------------
    Tim Zielke
    ------------------------------

    Original Message