IBM Sterling Transformation Extender

Sterling Transformation Extender

Come for answers, stay for best practices. All we're missing is you.


#Sterling
#Supplychain
 View Only
Expand all | Collapse all

WTX mapping and Datapower

  • 1.  WTX mapping and Datapower

    Posted 04/15/09 03:02 PM

    Originally posted by: dkbjr


    I'm still fairly new to Transformation Extender Design Studio and Datapower butI have created a map that transforms data from COBOL copybook format to XML. I tested it on the DataPower appliance and everything is working well. However, the vendor I am working with has requested that each "record" of the input data in COBOL copybook format be sent over to them as a single XML message. I am currently creating one XML file with all the information from the input file.

    Is the creation of single XML messages and the processing of them something that can be done? And if so, is it configured at the WTX Design Studio level, the DataPower level or both.

    Any help and/or examples would be greatly appreciated.

    Thanks,
    Don
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 2.  Re: WTX mapping and Datapower

    Posted 04/15/09 03:48 PM

    Originally posted by: repanzer


    If a WTX map takes in the COBOL file and outputs the XML, it’s a WTX fix.

    Let’s say my input has 10 records and my output is an EDI file with the three normal envelopes (ISA, GS, ST).

    In the WTX map on the highest level (interchange, ISA), if I build a functional map and I pull in one record of the ten on that functional map, I’m telling the map that each record equals one Interchange.

    But instead, if I pull the entire input file into the functional map at the ISA level, I’m saying that each file will equal one interchange.

    Same thing from the GS level ( considering I pulled the file into the functional map at the ISA level) ; if I pull in one record to the functional map, I’m saying that each record equals a functional group (GS level), and there will be 10 functional groups in one interhchange.

    Or, I can pull the entire file in to the GS functional map saying one file equals one interchange and one functional group (GS). Now that I’m ready to map the transactions, I pull one record in the functional map that creates each transaction (assuming that each record contains the info for one transaction.

    The same thing goes for your file to XML. You’re taking the entire COBOL copybook input and pulling the entire file at the highest level, saying to the map that you will get one XML file per COBOL file.

    But you want to pull records out of the COBOL file and pass them to the functional map at a higher level, saying that each record in the COBOL file equals **What even it equals in your XML file**.

    Where in your map that you pull the record instead of the file into a functional map depends on your file layouts.
    #IBM-Websphere-Transformation-Extender
    #DataExchange
    #IBMSterlingTransformationExtender


  • 3.  Re: WTX mapping and Datapower

    Posted 04/16/09 10:55 AM

    Originally posted by: dkbjr


    I was hoping that the creation of single XML messages wasn't going to be an issue and I more or less understand what you described and yes, each record will contain the information for one transaction.

    However, I'm still a little unsure on how to "pull records out of the COBOL file and pass them to the functional map at a higher level" .

    If my COBOL copybook input looks like this:

    01 DEALER-FILE.
    05 DEALER-RECORD.
    10 DEALER-CODE PIC X(05).
    10 DEALER-NAME PIC X(45).
    10 DEALER-ADDRESS-PRIM-SERVICE.
    15 DEALER-STREET-LINE1 PIC X(60).
    15 DEALER-STREET-LINE2 PIC X(34).
    15 DEALER-CITY PIC X(30).
    15 DEALER-STATE PIC X(02).
    15 DEALER-ZIP PIC X(09).
    15 DEALER-ZIP-PO PIC X(09).
    10 DEALER-PRIM-SERV PIC X(01).
    10 DEALER-FILLER PIC X(10).

    and my XML output looks like this:

    <dealerRequestMsg>
    <messageHeader>
    <source/>
    <transactionId/>
    <messageName/>
    <messageCreationTime/>
    </messageHeader>
    <dealerInfo>
    <dealerCode/>
    <dealerDBAName/>
    <dealerAddress>
    <city/>
    <postalCode/>
    <state/>
    <country/>
    <dealerStreetAddress/>
    <primaryAddress/>
    </dealerAddress>
    </dealerInfo>
    </dealerRequestMsg>
    How would I setup my map so that I am pulling a record at a time from the file and not the whole file?
    I apologize in advance if this is a very basic question, but this is still fairly new to me.

    Thanks for your help.

    Don
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 4.  Re: WTX mapping and Datapower

    Posted 04/16/09 01:55 PM

    Originally posted by: janhess


    The easiest way is to set your input card to burst mode with a burst size of 1. Have your output XML card set to sink and have a second output card with a rule (one item field) that PUTs the XML card to a unique file name. That way you read 1 record and output 1 file until you reach the end of the input.
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 5.  Re: WTX mapping and Datapower

    Posted 04/16/09 02:40 PM

    Originally posted by: repanzer


    janhess, do you often use burst to accomplish something like this? I've never used burst, thinking it was more for processing large amounts of data, and wasn't aware it could be used for mapping desired output. I always build the transactions as I mentioned (into one file), then pull each transaction into an f_map in a second output card, doing a "PUT" for each one, as you suggested.

    So if you have an input of File>Records(s), and set it to burst 1, it will run the entire map for each record? Meaning it will grab the first record, build output card 1, then output card 2, then start again with the next record, as opposed to not using burst and it builds output card 1 for all records, then output card 2, then end? Thanks!!!
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender
    #DataExchange


  • 6.  Re: WTX mapping and Datapower

    Posted 04/16/09 03:26 PM

    Originally posted by: janhess


    If you use burst mode it processes the requested number of input records then processes each output card for the same number. So where normally you would read all your input file, process all records for output1 and then all records for output2, if you have a burst of 1, it reads record 1 and processes output1 and output 2 then reads record 2 and processes output etc. That way you can avoid the need for the functional maps. When using database for output and you set the output card to commit by card, you can commit each record as it processes which can be useful if you need to reference a previously processed record.
    #IBM-Websphere-Transformation-Extender
    #DataExchange
    #IBMSterlingTransformationExtender


  • 7.  Re: WTX mapping and Datapower

    Posted 04/16/09 02:33 PM

    Originally posted by: repanzer


    Start WTX info;

    Assuming you have an input cobol file (file = top level in type tree) of 10 records and your XML output is a file (top level) with X number of XML groups, and each group has a header, X number of transactions, and a detail record;

    As I’m hoping you’re aware, any functional map will run for the number of instances of the type you pull with it.

    Looking at the attached picture “EachRecordGetsHeader.JPG”, I’m pulling in the record in the functional map f_MAP_XML, saying, run this functional map for each record in my input file. Being the header and detail record are part of the XML Group, we’re saying that each record will get a header, transaction, and a detail. So the map kicks off, it grabs the first record and runs f_MAP_XML for the that record, builds a header, a transaction, a detail record, then it goes back and grabs the second record, builds the header, trans, and detail, then goes back and grabs the third record, and continues until all input record are processed, so each record produces a header, trans, and detail.

    Looking at the picture “OneHeaderOnly.jpg” I pull in the file, NOT the record into f_MAP_XML, and being there is only one “file”, we’re saying run f_MAP_XML just once, so no matter how many input records are in that file, only one header and detail record will be produced. So the map kicks off, it grabs the entire file and runs f_MAP_XML. Then while in f_MAP_XML, there is another functional map on the transaction level that grabs each record and builds a transaction. For you final output, you will have 1 header, 10 transactions, and one detail record.

    End WTX info;

    The second example sounds like what you’re getting, when you want something closer to the first example.

    Still, if you have one input and one output card, even if you make the change and pull at the correct level, you’re still going to wind up with one output file. It will have multiple instances of the entire XML message as the vendor requested, but all the messages will be in the same file.

    Does your output currently looks like this (multiple dealerrequest, one for each record) ;
    <dealerRequestMsg> 1
    ....
    ....
    </dealerRequestMsg>
    <dealerRequestMsg> 2
    ....
    ....
    </dealerRequestMsg>
    <dealerRequestMsg> 3
    ....
    ....
    </dealerRequestMsg>
    Or does it look like this (one dealerRequestMsg, multiple loops in dealerrequestmsg) ;

    <dealerRequestMsg> 1
    ...<messageHeader>
    ..<source/>
    ..<transactionId/>
    ..<messageName/>
    ...<messageCreationTime/>
    ...</messageHeader>
    ...<dealerInfo> 1
    ......<dealerCode/>
    ......<dealerDBAName/>
    ......<dealerAddress>
    ......<city/>
    ......<postalCode/>
    ......<state/>
    ......<country/>
    ......<dealerStreetAddress/>
    ......<primaryAddress/>
    ......</dealerAddress>
    ...</dealerInfo> 2
    ...<dealerInfo>
    ......<dealerCode/>
    ......<dealerDBAName/>
    ......<dealerAddress>
    ......<city/>
    ......<postalCode/>
    ......<state/>
    ......<country/>
    ......<dealerStreetAddress/>
    ......<primaryAddress/>
    ......</dealerAddress>
    ...</dealerInfo>
    ...<dealerInfo>3
    ......<dealerCode/>
    ......<dealerDBAName/>
    ......<dealerAddress>
    ......<city/>
    ......<postalCode/>
    ......<state/>
    ......<country/>
    ......<dealerStreetAddress/>
    ......<primaryAddress/>
    ......</dealerAddress>
    ...</dealerInfo>
    </dealerRequestMsg>
    ?????

    How to continue will be dependent on that answer.

    I know how to fix it my way, but janhess's input may be a better way, a way that I am not familiar with (but am highly interested in).

    All in all, deciding what to pull into a functional map in attempts to create the desired output is undoubtedly one of the more challenging aspects of coding in WTX and even the most experienced programmer has to take a moment or two and figure this part of the mapping, prior to coding.
    #IBM-Websphere-Transformation-Extender
    #DataExchange
    #IBMSterlingTransformationExtender


  • 8.  Re: WTX mapping and Datapower

    Posted 04/16/09 02:33 PM
      |   view attached

    Originally posted by: repanzer


    attached pictures
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender

    Attachment(s)



  • 9.  Re: WTX mapping and Datapower

    Posted 04/16/09 06:12 PM

    Originally posted by: dkbjr


    You are correct, I am currently getting the second example ( one dealerRequestMsg, multiple loops in dealerrequestmsg )and want something closer to the first example.

    And actually, I had previously set the input card to burst mode with a burst size of 1 and it gave me exactly what I needed ( multiple dealerRequestMsg, one for each record ) except everything was in one output file. What I didn't know how to do was to split it so each message would go to a seperate output file.

    It looks like janhess's input may be the way to go ( actually that helps me with another project where the vendor is looking for the information in batches of 750 ). And will give it a try tonight and update you with my results.

    I am also interested in seeing how you would of fixed it as well, if you wouldn't mind.

    Thanks again,
    Don
    #DataExchange
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender


  • 10.  Re: WTX mapping and Datapower

    Posted 04/16/09 06:34 PM

    Originally posted by: repanzer


    Janhess, thanks for the info. I should probably know that by now.

    Dkbjr,

    Just as we both mentioned, in order to move these pieces of data, you need a second output card.

    If you’re to go with the burst mode, you need an output card with one rule which does a PUT(“FILE”, /dir/filename.txt, PACKAGE(INPUT). Make sure the filename has a date/time stamp in it so you get different files for each one, the suggestion to use MMDDYYHHMMSS0-9. 0-9 is the millisecond. So it will grab a record, map the XML, put it in the file, then start from the top.

    In event you had mapped the data without burst, and had your type tree looking like;

    XMLFile
    ….Transaction
    ……Dealermsg
    ……/dealermsg
    ….Transaction
    ……Dealermsg
    ……/dealermsg
    Etc…

    And now you had one file with multiple Transaction/dealermsg, then you need your second output card to be a file with multiple record.

    File
    …Record(s).
    …..blobfield

    Now that you have this and just in case you’re not aware, while in map designer, left-click and hold you first output card in the little area where it says #1, just above output | rule, and drag it away until the curser changes to the plus sign. Relase the left click and now you should have a duplicate image of the output card.

    In a functional map (f_PUT_FILE) on the record level for your second output card, pull the Transaction type from the duplicate image of the first output card (that why we created the duplicate image just before, so you can use pull the types from it into a map rule on a different output card). Now that functional map (f_PUT_FILE) will run once for each transaction you built, one by one. In the functional map on the blob field, do the PUT(“FILE”). Just as the functional map runs for each transaction in your first output, the PUT statement will, putting just one transaction at a time. Remember to use the millisecond so you don’t over write any files on the PUT statement. Also, just incase it doesn’t allow you to put the date time stamp using a DATETIME function directly in PUT(FILE) statement, put another blob field in the record of the second output card, build the date/time stamp in the first field, and do the PUT in the second, using the first field in the output file name.
    #IBM-Websphere-Transformation-Extender
    #DataExchange
    #IBMSterlingTransformationExtender


  • 11.  Re: WTX mapping and Datapower

    Posted 04/17/09 10:43 AM

    Originally posted by: dkbjr


    Well, I have good news and bad news.

    First the good news, I went with the burst mode, created the 2nd output card and it worked great. It created a file for each transaction.

    The bad news is, it only works when I'm using the WTX Map Runtime. When I'm in the DataPower Map Runtime, it gives me the following error message "Target Adapter is not valid for DataPower". So it looks like I cannot use "SINK" as the Target Adapter if I am deploying the map to DataPower ( which is what I will be doing ).

    Do we have another way of doing this ?

    You had mentioned that you knew how to fix it your way, can you provide some more detail as to what that was.
    #IBMSterlingTransformationExtender
    #DataExchange
    #IBM-Websphere-Transformation-Extender


  • 12.  Re: WTX mapping and Datapower

    Posted 04/17/09 01:19 PM

    Originally posted by: repanzer


    If it is just that DataPower does not recognize SINK, you can change it to FILE. SINK just holds the info in memory during the map execution, while FILE will write it to a file. Give that a shot and let us know how it worked out.
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender
    #DataExchange


  • 13.  Re: WTX mapping and Datapower

    Posted 04/17/09 02:03 PM

    Originally posted by: dkbjr


    I had tried that and it gave me a "Rule references unkown function: PUT" error during compilation. After further investigation I found that the PUT function is not supported by the DataPower appliance.

    Any thoughts?
    #DataExchange
    #IBM-Websphere-Transformation-Extender
    #IBMSterlingTransformationExtender


  • 14.  Re: WTX mapping and Datapower

    Posted 04/17/09 06:05 PM

    Originally posted by: repanzer


    No easy ones. I have no knowledge of datapower what-so-ever, so I'm limited to WTX info.

    Instead of having one map with your input and output as you have now, you may need to build a control map that has the cobol input, and in a rule in the output card, you execute a run map, passing one record at a time as the input, and the output of the xml. You override the output card setting with a unique filesname (date/time) and if you have 10 record in your input, you run the runmap 10 times and create 10 output files.

    Take your current map, save it under a different name, point you're input type tree to the record level, build the map, work out the errors (the names will no longer match what you have in the map rules**), and then you'll have your run map. Building the runmap statment will be your biggest challange. It's no challenge for experienced WTX programmers, but it takes some getting used to for beginners.

    It may sound complicated, and it may be for what you're used to, but that may work, because you won't use a PUT.
    • anytime you change the input type, it messes up all the map rules you've already coded, but this can be more easily be fixed with find/replace.

    What version are you on - WTX?
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender
    #DataExchange


  • 15.  Re: WTX mapping and Datapower

    Posted 04/20/09 10:06 AM

    Originally posted by: dkbjr


    I am using WTX version 8.2 but it looks like the RUN function is not supported by DataPower either.

    I found some documentation for the "for-each" action on the DataPower appliance which looks like I may be able to use to create the individual output files. If I can use this along with the burst size of one than I should be good to go.

    Will try this today and provide an update on the outcome.

    Thanks,
    Don
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender
    #DataExchange


  • 16.  Re: WTX mapping and Datapower

    Posted 05/19/09 09:57 AM

    Originally posted by: dkbjr


    Just an update and to close this thread, I was able to get this done using the for-each action on the DataPower Appliance.

    Thanks repanzer and janhess for your help!
    #DataExchange
    #IBMSterlingTransformationExtender
    #IBM-Websphere-Transformation-Extender