Planning Analytics

Planning Analytics

Get AI-infused integrated business planning

 View Only

  • 1.  Turbo Integrator - Read multiple file at once

    Posted Wed January 15, 2020 03:55 AM
    Hi all,

    Is it possible to read multiple files at once when using turbo integrator for cube update. I have 24 csv files differentiate by filename source_yyyymm.csv (source_201901.csv, source_201902.csv, etc..). Some solutions I found while browsing is create 24 TI process or using paw. However I prefer using TI for data manipulation. I was thinking whether TI can read multiple files using wildcard characters?

    Thanks in advance :)

    regards

    ------------------------------
    ida lasim
    ------------------------------

    #PlanningAnalyticswithWatson


  • 2.  RE: Turbo Integrator - Read multiple file at once

    Posted Wed January 15, 2020 04:13 AM

    Hi Ida, 

    How is the structure of the files ? Is it the same for each file ? 

    You can do 2 processes :

    The first is the import of the file, the name could be parametric. You have to use "DatasourceNameForServer = sFilePath;" to define the file you import, 

    And another process which run 24 times the first one with "ExecuteProcess"' and the name of the file as parameter

    Regards, 



    ------------------------------
    Corentin Gombert
    ------------------------------

    Original Message


  • 3.  RE: Turbo Integrator - Read multiple file at once

    Posted Thu January 16, 2020 03:56 AM
    Hi Ida,

    Corentin's solution is one of the most used but in case you need to run the process only once because of other updates in metadata or data, then you can use a subprocess in the prolog tab which merges the files to one, which can be then uploaded.

     ExecuteProcess('System - Merge Files');​

    If you know the number and names of the files then you can hardcode it. Alternatively, you could use the WildcardFileSearch() function to collect all files meeting the set criteria.

    ### Merge Files ###
cCommand = 'cmd /c copy /b ' | pFile1 | '+'  | pFile2 | '+' | pFile3 | ' ' | pMergedFile;
ExecuteCommand(cCommand,1);​

    There are also some prerequisites which are important:
    • the merge needs to be executed in a subprocess because the newly created file needs to be "closed" / "released" before import
    • the to be merged source files need to have the same structure and encoding before the merge. You can also use Powershell instead of CMD and convert the files before merging.
    Cheers,

    ------------------------------
    Tomasz Brzoza
    Linde AG
    Munich
    ------------------------------

    Original Message


  • 4.  RE: Turbo Integrator - Read multiple file at once

    Posted Thu January 16, 2020 03:41 AM
    Hello! if the number of file is variable, you can also use the function WildcardFileSearch based on a standard name of the file.
    Regards,
    Cécile

    ------------------------------
    DESCHARD Cécile
    ------------------------------

    Original Message


  • 5.  RE: Turbo Integrator - Read multiple file at once

    Posted Thu January 16, 2020 11:05 AM
    Hello Ida-

    I'm not sure from your question if you want to multi-thread the loads but if you do here's a simple set of steps you could follow.

    1. Create a "wrapper" TI process
    2.     Use the TI function Execute Command function (in the Prolog) to create the list of the files you want to load.
    2. Use that list as the data source  
    3. Use the Run Process command (on the metadata tab) to execute each of the data loads on a separate thread.

    Chris

    ------------------------------
    Chris Courim
    ------------------------------

    Original Message