Planning Analytics

 View Only
Expand all | Collapse all

Prevent a TI Process from Running Multiple Instances

  • 1.  Prevent a TI Process from Running Multiple Instances

    Posted Thu October 10, 2024 10:31 AM

    Hey everyone,

    I'm trying to figure out how to prevent a TI process from running if another instance of the same process is already in progress. I want to avoid data-locking conflicts, as I've experienced situations where processes end up waiting for each other and automatically stopping and starting endlessly.

    Does anyone have any tips or strategies for checking if a TI process is currently running before allowing it to execute again? Any insights would be greatly appreciated!

    Thanks!



    ------------------------------
    Asgeir Thorgeirsson
    ------------------------------


  • 2.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Thu October 10, 2024 11:25 AM

    Hello Asgeir,

    In prolog, i use 

    Synchronized( GetProcessName() ) ;

    Regards,

    Philippe



    ------------------------------
    Philippe CHAMPLEBOUX
    ------------------------------

    Original Message


  • 3.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 06:01 AM

    Thanks @Philippe CHAMPLEBOUX for the suggestion!

    Using Synchronized(GetProcessName()); seems like it could help, but I'm curious: does it completely block a second instance of the process from starting until the first finishes, or does it just delay it?

    I want to avoid an endless cycle of processes waiting for each other, so if it can handle that scenario, it sounds like a great fix.



    ------------------------------
    Asgeir Thorgeirsson
    Financial Solutions Engineer
    Icelandair
    Reykjavik
    +3548930750
    ------------------------------

    Original Message


  • 4.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 08:14 AM

    Hi asgeir,

    When a firsqt occurence run and a second (or third, or .......) subsequent ocurrence are "waiting for release lock".

    When the first occurence finish, the "IXC lock is released" and a other occurence can run (aleatory, the second, third, .....)

    I purpose you a test :

    create a ti process with 

    Synchronized(GetProcessName());

    Sleep (60000);

    execute this process and fews seconds execute again this process.

    With tm1top (or workspace administration threads, ) you can see the reality (lock, waiting, etc ....)

    Regards,

    Philippe



    ------------------------------
    Philippe CHAMPLEBOUX
    ------------------------------

    Original Message


  • 5.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Thu October 10, 2024 11:26 AM

    Hi Asgeir,

    We use a cube with two dimensions, "process" dimension with a list of process names and a numeric flag dimension.

    At the very begginning of the process, we check for that flag. If it is off, then we turn it on and the process runs as normal. If it is on, then the process brakes to the epilog and ends. 

    For us, a user must turn the flag off for the process to run again.

    Hope this helps.

    Regards.



    ------------------------------
    Jose Luis Senas
    Data Analyst
    Kimberly-Clark Mexico
    Mexico City
    ------------------------------

    Original Message


  • 6.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 08:36 AM

    Hi Jose,

    Be careful with reading and writing cell values as flags to orchestrate Turbo Integrator processes!  It's important to understand that a TI process does not change data in cubes until the TI process has successfully completed.  All data changes from the TI process are only committed and readable by other processes after the TI process is complete.  This means if you start a second instance of the TI process while the first instance is running it will not see the flag value as being set. 

    Using the CubeSaveData Ti function to commit a flag value mid process is not recommended (but I've seen this technique used successfully).

    You might want to consider a wrapper TI process that reads and writes the flag cell value.  The wrapper process can use the RunProcess function to start that actual worker processes after reading or writing the flag value.  This should ensure the flag value is visible to other processes that start after the wrapper process has completed (the wrapper process should complete very quickly).



    ------------------------------
    Stuart King
    Product Manager
    IBM Planning Analytics
    ------------------------------

    Original Message


  • 7.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 09:23 AM

    Thanks for pointing this out @STUART KING

    I have another concern regarding the use of flags in TI processes.

    In a TI import process, I use a flag with the process name to disable conditional feeders in the target cube during data insertion. My intent is to avoid performance issues since recalculating the feeders for each record inserted can be slow. So, I disable the feeders while inserting data and re-enable them at the end, followed by a recalculation using:

    CubeProcessFeeders('TargetCube');

    I'm using conditional feeders like this:

    

    [dim:'members'] => DB(
    if(DB('RunProcess', 'ProcessName', 'Active') = 0, 'TargetCubeName', ' '), 
    !DIM1, !Dim2, ..., !DimX
);

    This setup disables feeders when the process is running and re-enables them once it's complete. I'm wondering if this approach is optimal or if there's a better way to handle feeder management during large imports to avoid performance degradation.

    Any suggestions or feedback would be appreciated!



    ------------------------------
    Asgeir Thorgeirsson
    Financial Solutions Engineer
    Icelandair
    Reykjavik
    +3548930750
    ------------------------------

    Original Message


  • 8.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 10:34 AM

    If only one process is in this case, Synchronized( GetProcessName() ) ; is heavy. Using a cube is not a good idea as it will put locks on it, plus as I test, the write in the cube is not alway detected. The more simple is to write a file somewhere and delete at the end of the process. If the file exist you stop the process or wait in it that the file is not there. Wirte it by Executcmd to not have to wait the end of the prolog to have the file.

    Regards  



    ------------------------------
    Frederic Arevian
    ------------------------------

    Original Message


  • 9.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 11:21 AM
    Edited by Vlad Didenko Fri October 11, 2024 05:05 PM

    The best option depends on your specific requirements. If you need to run the process several times one by one, then using the "Synchronized" option is recommended. In other cases, I have used a system cube to store the statuses and write logs using "CubeSaveData" without any problems. Another simple solution is to use a text file as a flag. You just need to manage the logic to ensure the file is deleted if the process fails, which can be done in a wrapper process.

    When performing extensive bulk cube updates, you may want to consider detaching the cube rules (using RuleLoadFromFile) and disabling cube logging. Then you can re-enable the rules and logging once the load is complete.



    ------------------------------
    Vlad Didenko
    Founder at Succeedium
    TeamOne Google Sheets add-on for IBM Planning Analytics / TM1
    https://succeedium.com/teamone/
    Succeedium Planning Analytics Cloud Extension
    https://succeedium.com/space/
    ------------------------------

    Original Message


  • 10.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 02:23 PM
    Edited by Asgeir Thorgeirsson Fri October 11, 2024 02:29 PM

    Thanks for the input @Vlad Didenko! 

    I use SaveDataAll cautiously-just once per night, right before the daily backups-for two reasons: it can cause lock contention, and it affects users' ability to roll back their latest manual inputs.

    I'd like to explore or learn more about your suggestion to detach cube rules with RuleLoadFromFile during bulk updates. It sounds like a great way to optimize performance. Could you share more details or examples of how you've used this approach?

    Appreciate the insights!



    ------------------------------
    Asgeir Thorgeirsson
    Financial Solutions Engineer
    Icelandair
    Reykjavik
    +3548930750
    ------------------------------

    Original Message


  • 11.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Fri October 11, 2024 03:58 PM

    Apologies for the confusion earlier, I was thinking about CubeSaveData instead of SaveDataAll. You're absolutely right that we want to avoid unnecessary locking.

    To handle bulk data updates efficiently, you can temporarily remove the cube rule with CubeRuleDestroy before processing updates, and then restore it afterward using RuleLoadFromFile.

    Disabling and re-enabling cube logging will also improve performance during the updates.

    I would recommend using a wrapper process where you:

    1. Disable cube logging with CubeSetLogChanges(cub, 0) to reduce overhead during data processing.
    2. Use a CMD or PowerShell command to copy the current RUX file to back it up.
    3. Remove the cube rule with CubeRuleDestroy(cub).
    4. Run your data update process using ExecuteProcess().
    5. Once the processing is complete, restore the cube rule with RuleLoadFromFile(cub, 'PATH_TO_COPIED_RUX_FILE').
    6. Finally, re-enable cube logging with CubeSetLogChanges(cub, 1).

    Of course, you have to keep in mind the changes will not be available in the Transaction log if you disable logging, so it may not be an option for you.

    I hope this helps!



    ------------------------------
    Vlad Didenko
    Founder at Succeedium
    TeamOne Google Sheets add-on for IBM Planning Analytics / TM1
    https://succeedium.com/teamone/
    Succeedium Planning Analytics Cloud Extension
    https://succeedium.com/space/
    ------------------------------

    Original Message


  • 12.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Sat October 12, 2024 07:54 AM
    Edited by Asgeir Thorgeirsson Sat October 12, 2024 08:09 AM

    Thanks for the input! 

    Given the complexity of handling these scenarios, I'd like to challenge the IBM PAW/TM1 development board to provide a built-in, robust, and easy-to-implement solution for managing bulk data updates. A function call to disable all feeders, calculations, and logging during updates would greatly simplify the process.

    I’d suggest a function like:

    • CubeRuleActive(cube, 0) to disable feeders, calculations, and logging, and
    • CubeRuleActive(cube, 1) to re-enable them once updates are complete.



    ------------------------------
    Asgeir Thorgeirsson
    Financial Solutions Engineer
    Icelandair
    Reykjavik
    +3548930750
    ------------------------------

    Original Message


  • 13.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Wed October 16, 2024 09:40 AM

    Small point, but I would avoid CubeSetLogChanges(...) as it requires the process to acquire a significant amount of locks and can be contentious. To avoid the lock contentions use CellPutS('No', '}CubeProperties', ..., 'Logging');



    ------------------------------
    Ryan Clapp
    ------------------------------

    Original Message


  • 14.  RE: Prevent a TI Process from Running Multiple Instances

    Posted Tue October 15, 2024 07:05 PM

    Hi Asgeir, 

    Have you tried using the NumericGlobalVariable('variablename') to accomplish this?



    ------------------------------
    Craig Sawers
    ------------------------------

    Original Message