Planning Analytics

 View Only
  • 1.  Change record terminator in TI

    Posted Fri February 04, 2022 09:04 AM

    A left-field one today.  I have not seen a way to change the record terminator in TI when dealing with flat files but hoping there is a way.

    I have an issue where some of my data contains a Linefeed, Char 10 (Hex 0A).
    Probably from an upload someone did from Excel where Alt+Enter was used to split the value across multiple rows for readability.

    TI seems to pick up any of Hex 0A, 0D or 0D0A as a valid row terminator thus splitting the record into two and creating havoc.

    Yes, this needs to be sorted in master data, at source but curious to know if there is a way to trap for this in TI or specify that 0D alone is the record terminator.
    Short-term solution may be to use SED or Python to strip 0A's

    Anyone have any ideas on how to change the row terminator?



    ------------------------------
    George Tonkin
    ------------------------------

    #PlanningAnalyticswithWatson


  • 2.  RE: Change record terminator in TI

    Posted Fri February 04, 2022 10:23 AM
    Hi George,
    To understand this issue better what did you setup as youe source separator?
    Best regards,


    ------------------------------
    Svetlana Pestsova
    IBM Planning Analytics Product Manager
    ------------------------------



  • 3.  RE: Change record terminator in TI

    Posted Fri February 04, 2022 10:30 AM
    Hi Svetlana,
    The file is a text file containing pipe or vertical bar, ascii character 124. The fields are fine but where the record contains a CR or LF in the record, the record is split in multiple parts as TM1 sees these as record terminators.

    Product description field below shows issue:

    Essentially TI would need to ignore CR or LF inside quotes, much the same way SQL, Excel etc. does otherwise I need a way to specify the record terminator i.e. CRLF or LF only or something else.

    ------------------------------
    George Tonkin
    ------------------------------



  • 4.  RE: Change record terminator in TI

    Posted Fri February 04, 2022 03:46 PM
    Hi George,
    HEX 0A is a line feed, 0D and 0D0A are carriage return line feed,
    they all serve the same function: to separate lines. 
    There is no way to do the distinction on the TI side. 
    Maybe you could do script searches, some clients searching the input data for other sort of characters to assure the data integrity but I've not cross the line feed.

    The only solution is to ask users not to do Alt+Enter in Excel, as any script will not be able to know when the row ends, if it ignores these characters.

    Best regards,

    ------------------------------
    Svetlana Pestsova
    IBM Planning Analytics Product Manager
    ------------------------------



  • 5.  RE: Change record terminator in TI

    Posted Mon February 07, 2022 04:04 AM
    By following this thread I gained a pack of knowledge. Happy to be a member this community!

    ------------------------------
    Amanda Nunes
    ------------------------------



  • 6.  RE: Change record terminator in TI

    Posted Tue February 08, 2022 02:16 AM
    Hi George, 
    The problem is as old as information systems I believe, when some obscure developer decided ASCII would not use the same line terminators as EBCDIC or Unicode...
    I've written several scripts over the time to get rid of those pesky line terminators in the middle of text files, in C then perl then powershell. 
    Shoot me an email and I'll send you the powershell version ;)
    Cheers,

    ------------------------------
    Laurent Henssien
    C L A R I T Y
    https://www.clarity.consulting
    ------------------------------



  • 7.  RE: Change record terminator in TI

    Posted Tue February 08, 2022 02:20 AM
    Hi Laurent,

    Thank you for the reply and the offer. For now I have a script too that basically swaps 0D0A for Character 255 then strips any remaining 0A or 0D's.
    Once that is done, Character 255's are replaced again with 0D0A.

    Will still be pushing back to the master data team to sort things out in SAP.

    ------------------------------
    George Tonkin
    ------------------------------