IBM Sterling Transformation Extender

Sterling Transformation Extender

Come for answers, stay for best practices. All we're missing is you.


#Sterling
#Supplychain
 View Only
Expand all | Collapse all

Create a type tree for variable length component

  • 1.  Create a type tree for variable length component

    Posted 12/28/08 11:55 PM

    Originally posted by: SystemAdmin


    Hello experts,

    I have a question about creating type tree in WTX 8.2 to accommondate my data file. Below is an overview of the characteristics of my data source file:

    • it contains serveral lines with the same definition.
    • each line contains 3 columns. and each column is fix in size. column 1, 10 characters long, column 2, 20 characters long, column 3, 15 characters long.
    • if data for a column is less than the maximum size, it's padded with space.
    • One exception for the last column, if it's data is less than its maximum size, it wont be padded with space and kept unchanged. So the total length of the line is less than the sum of each column's size.

    with the definition above, Can you help me to contruct a type tree? i've tried creating a group with 3 components, and set the group syntax to fix, but it doesn't work because of the exceptional case in the data definition.
    Thanks in advance,

    mrpc_cambodia
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 2.  Re: Create a type tree for variable length component

    Posted 12/29/08 12:14 AM

    Originally posted by: thisisgaurav


    As far as I understand, you want the length of the last column equal to the length of the incoming data for this last column. If that is the case then you can still use fix length for the last column with <space> padding and inside the map rule you can use TRIMRIGHT() function to remove extra padded spaces for this field.

    In an extreme case if the incoming data for this 3rd column itself contains spaces (as data) then instead of padding the fix length with <spaces> use some special characters such as $*(!^%# etc and then inside the map rule use TRIMRIGHT(last column,'special character') ..... to remove that to have 3rd coulmn with length equal to that of data.

    Please let me know if this is what you were looking for.
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender
    #DataExchange


  • 3.  Re: Create a type tree for variable length component

    Posted 12/29/08 01:52 AM

    Originally posted by: SystemAdmin


    Thanks for your comment,

    My question is not clear enough and am sure it confuses you. let me clarify:

    The length of the actual data for the last column that is fed to WTX is less than the maximum size which is 15. you are right, i could pad space to the data and trim it back within WTX, but the problem is the data is sent from customer side, not from my own application so i don't have any control over the content of this data.

    The customer sends data to my company, so we forward it to WTX to translate, and when the length of the last column is less than its maximum size, WTX reports "Invalid input data". So i need a solution for this case.
    Thanks once again, your comment is highly appreciated.

    mrpc_cambodia
    #DataExchange
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender


  • 4.  Re: Create a type tree for variable length component

    Posted 12/29/08 02:11 AM

    Originally posted by: thisisgaurav


    Agreed, you don't have control over user's data. But you can definitely pad the 3rd column with a character you are sure that the incoming data will never have. For example "" shall not bein the user data for this 3rd column. So, if you are sure that "" is a chatacter that was added by you inside the type tree then .. its easy to extract back the original user data in WTX.

    If you think I have still not understood your question then please give an example. I definitely try to get a solution.
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 5.  Re: Create a type tree for variable length component

    Posted 12/29/08 02:27 AM

    Originally posted by: SystemAdmin


    Thanks again for your quick response.

    because i wanted my question to be simple and easy to understand, so i only mentioned that only the last column that its data's length could be less than it's maximum size. Actually the 2nd column or the first column, its data's length could also be less than its maximum size. Ok below is a sample data content that illustrates all the possible cases of the incoming data:

    Column 1 Column 2 Column 3
    --------- ------------------- --------------

    AAAAAAAAAABBBBBBBBBBBBBBBBBBBBCCCCCC
    AAAAAAAAAA CCCCCCCCC
    AAAAAAAAAABBBBBBBBBBB CCCCCCCCC
    AAAAAAAA BBBBBBBBBB
    AAAAA
    So can you show how should i create a type tree that is capable of representing this data? if you stick with the idea that i should pad space to the data, can you tell me from where can i do this? should i be within WTX? am really a newby to this problem.
    Thank you,

    mrpc_cambodia
    #DataExchange
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender


  • 6.  Re: Create a type tree for variable length component

    Posted 12/29/08 02:32 AM
      |   view attached

    Originally posted by: SystemAdmin


    Due to the text format of this forum, the structure of my sample text is changed. So i attach a sample text file.
    #IBM-Websphere-Transformation-Extender
    #DataExchange
    #IBMSterlingTransformationExtender

    Attachment(s)



  • 7.  Re: Create a type tree for variable length component

    Posted 12/29/08 10:00 AM

    Originally posted by: repanzer


    If I'm reading this correctly....

    You can't say the fields/records/file is fixed, and have a varible lenght field. Either it's fixed, or it's not.
    If you type tree says it's fixed, but the customer is sending varible lenght records, you will have to change the type tree to fix the definition of the data you're receiving.
    But I have a question before anything;

    If it is supposed to be fixed and they are sending varible lenght fields, how can you determine where one field starts and one field ends?

    In example, just say they are sending you a purchase order number and an invoice number and the fields are both defined as 5 bytes.

    PO number= 11111
    Invoice number = 22222

    so the file looks like this;

    1111122222

    And that works fine. But in the next example, the PO number is only 4 bytes. You're saying the data would look like this, correct?
    111122222
    Is that correct? Because if it is, you may have a big problem. If you yourself can not distinguish where one field ends and one field starts, the map/type tree can not either.

    So before we continue, can you answer that question?

    If the field you're worried about is the last field, it would be workable, but if it's the first or second field, you may have the problem.
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 8.  Re: Create a type tree for variable length component

    Posted 12/29/08 10:48 AM

    Originally posted by: SystemAdmin


    Thank repanzer for your participation.

    actually it's a rather complicated data format. for my example, my data has 3 columns. and when i say either column 1, 2, or 3 can have variable length data, indeed only one of these 3 columns can have variable length data. A column can have variable length data only when the collumns that follow are empty including space.

    for example, if column 1 has variable length data then the whole line is like the below:

    =====Column 1=====|====Column 2=====|====Column 3=====
    AAAAAAAAAAA

    if Column 2 has variable length data, then the whole line is like below:

    =====Column 1=====|====Column 2====|=====Column 3=====
    AAAAAAAAAAAAAAAAAABBBBBBBB

    Thank you,

    mrpc_cambodia
    #IBM-Websphere-Transformation-Extender
    #DataExchange
    #IBMSterlingTransformationExtender


  • 9.  Re: Create a type tree for variable length component

    Posted 12/29/08 12:55 PM

    Originally posted by: repanzer


    Ok; the issue is the that file does not retain trailing spaces. That's it.

    Are they FTPing the file to you, because if so, there is a setting that would retain the spaces.
    You can run the file through a quick map, in which the only thing it does, is pad each record to the fixed length, and keep the translation type tree as it is. If you do that, just have the input as:

    File
    ..Record(s)
    ....Blob(min = 0, max = blank)
    Output defintion as:

    File
    ..Record(s)
    ....Blob(min = 0, max = 45)

    Run a functional map at the record level, and drag the input blob into the output blob, and now you truly have a fixed record for the translaction map.
    The other way to do it would be to use partitioning, but before we go there, let me ask; can the data look like this;

    =====Column 1=====|====Column 2====|=====Column 3=====
    AAAAAAAAAAAAAAAAAABBBBBBBBxxxxxxxxxCCCCCCCCCCCCCCCCCCC

    x = SPACE

    ???
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 10.  Re: Create a type tree for variable length component

    Posted 12/29/08 08:40 PM

    Originally posted by: SystemAdmin


    Yes, you are right, the file doesn't retain trailing space.

    The data could come like the below 2 lines:

    =====Column 1=====|====Column 2====|=====Column 3=====
    AAAAAAAAAAAAAAAAAABBBBBBBBxxxxxxxxxCCCCCCCCCCCCCCCCCCC

    =====Column 1=====|====Column 2====|=====Column 3=====
    AAAAAAAAAAAxxxxxxxBBBBBBBBxxxxxxxxxCCCCCCCCCCCCCCCCCCC

    I have over a hundred different maps which has this space problem, can you recommend me the best way to solve it?
    Thank you,

    mrpc_cambodia
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 11.  Re: Create a type tree for variable length component

    Posted 12/30/08 12:54 AM

    Originally posted by: thisisgaurav


    It appears from all this that there is no way you can say where column 1 end and where column 2 starts same with column2 -column3.

    The only solution is, you ask your customer to either have a FIXED length columns(padded with spaces if the data length is less than column length) OR have a DELIMITER between column, otherwise it will be impossible to solve this proble.

    We could have helped you, if this problem was only on the last column, where we could have easily recognised where your last column if a record ends and where first the column of the next record starts by a delimiter (such as <NL>, <CR><LF> etc.)
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 12.  Re: Create a type tree for variable length component

    Posted 12/30/08 01:35 AM

    Originally posted by: SystemAdmin


    No, you don't get my point. Among the 3 columns, only the last column will have variable length data. the other 2 columns will have spaces padded to the maximum size if its actual data is less than the maximum size.

    so if only column 1 has data then the whole line is like the below:

    ====Column 1====
    AAAAAA<NL>

    if column 1, and 2 has data then:

    ====Column 1====|===Column 2====
    AAAAAAxxxxxxxxxxBBBBBBB<NL>

    Note: x means space.
    Thank u,

    mrpc_cambodia
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 13.  Re: Create a type tree for variable length component

    Posted 12/30/08 04:27 AM

    Originally posted by: thisisgaurav


    In the examples below:
    ====Column 1====
    AAAAAA<NL>

    if column 1, and 2 has data then:

    ====Column 1====|===Column 2====
    AAAAAAxxxxxxxxxxBBBBBBB<NL>
    where are columns 1 and column 2 padded with spaces ? I can see a <NL> immediately after first/second column. That shouldn't be the case. it should have been something like this:
    ====Column 1====
    AAAAAAxxxx<NL>

    if column 1, and 2 has data then:

    ====Column 1====|===Column 2====
    AAAAAAxxxxxxxxxxBBBBBBBxxxxxxxxxxxxx<NL>
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 14.  Re: Create a type tree for variable length component

    Posted 12/30/08 04:53 AM

    Originally posted by: SystemAdmin


    Thank everyone for your participation and helpful comment.

    I think the best solution is the ensure that each column has space padded to fill its maximum length. Then i can assign the group and item to be fixed size.

    Any further comments are welcome.
    Thank you,

    mrpc_cambodia
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender
    #DataExchange


  • 15.  Re: Create a type tree for variable length component

    Posted 12/30/08 10:38 AM

    Originally posted by: repanzer


    darakok, are you in Cambodia?!?!?
    There's a few ways you can fix this.
    The first solution I mentioned above, where you have a map in front of the other maps that pad the records before passing to the translation map. We used this method in my last job where files that were FTPed from the mainframe lost the training spaces in each record.

    Another is to use partitioning.

    In the partition, you say the record can be either of 3 formats.

    Format1
    File
    ...records(s)
    ....Field1 (0 min, 10 max) padded with spaces
    ....Field2 (0 min, 10 max) padded with spaces
    ....Field3 (0 min, 15 max) not paddded

    Format2
    File
    ...records(s)
    ....Field1 (0 min, 10 max) padded with spaces
    ....Field2 (0 min, 10 max) not padded

    Format3
    File
    ...records(s)
    ....Field1 (0 min, 10 max) not padded

    The problem with this solution is you have "hundreds of maps with the spacing issue". Changing the input type tree to use partitioning will require you to update all those maps, because you need a functional map for each of the formats under the partition. So your output rule will look like this;

    EITHER( f_map1(Format1), f_map2(format2), f_map3(format3)).

    f_map1, f_map2, and f_map3 will need to be built.
    Another is; you can define the field in the input record as just a blob of text instead of 3 fields.

    File
    ...Records(s)
    .....Blob_Of_Text(min 0, max 45)
    Then in your output rules, you say;

    Output_For_Field1= TRIMRIGHT( LEFT(BlobOfText, 10) ," ")
    Output_For_Field2= TRIMRIGHT( MID (BlobOfText,10,20) ," ")
    Output_For_Field3= TRIMRIGHT( MID (BlobOfText,30,15) ," ")

    Finally, assuming that the trailing spaces are lost in the tranmission of the data, and that the customer is willing to update their system, you can ask them to put a certain character before the record terminator. This will ensure the each record is 45 bytes when you get it;

    =====Column 1=====|====Column 2====|=====Column 3=====
    AAAAAAAAAAAxxxxxxxBBBBBBBBxxxxxxxxxCCCCCCCxxxxxxxxxxxx&
    Other then the first solution I mentioned, the other methods in this post require you to update all those hundred of maps (from what I can tell).

    Putting a map in front of the translation maps that adds the trailing spaces that are lost, would probably be the best way, because you'll only have to create a new, simple map, and not change the hundreds of existing maps. I would avoid re-building teh hundreds of maps at all cost, and just build one map to fix the data coming in before it gets translated.
    good luck
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 16.  Re: Create a type tree for variable length component

    Posted 12/30/08 08:27 PM

    Originally posted by: SystemAdmin


    hi repanzer,

    your first solution is clearly the best option.

    Thank you so much for your clear explanation. And yes, I am in Cambodia.
    Happy new year 2009.

    mrpc_cambodia
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender
    #DataExchange


  • 17.  Re: Create a type tree for variable length component

    Posted 12/30/08 01:40 AM

    Originally posted by: thisisgaurav


    ok if that is the case then make your last column Min = 0 and Max = 15 and use a terminator as <NL> or <CR><LF>. Make sure youre last column is not FIXED, but the first and second column are FIXED and are padded with <SP>
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender