TRIRIGA

 View Only

 Question about scheduling import/export of tab separated flat file

HIROYA OGIHARA's profile image
HIROYA OGIHARA posted Wed August 20, 2025 02:23 AM
I have a question about scheduling the following processes:
      - Import BO from a tab separated flat file.
      - Export BO to a tab separated flat file.
 
We are considering using the following TRIRIGA functions to these import/export processes:
      Import (Tab separated flat file --> BO): 
         - Data Integrator or Integration Object 
      Export (BO --> Tab separated flat file): 
         - Integration Object
                                     
Data Integrator has a scheduling function (batch upload), however Integration Object does not have a scheduling function.
We thought we may be able to use Job Scheduler to schedule import/export processes of Integration Object. However it seems that the Job Scheduler can schedule only ETL Jobs, that can output BO data to BO (Fact table(triMetricFact)(Kettle Transformation), Classifications(Rebuild Hierarchy)) , not to a flat file.
 
My question is:
 
  How can we schedule import/export of Integration Object?
I'd really appreciate it if you answer my question.
Regards, Ogi
Bradley Stockdale's profile image
Bradley Stockdale posted Thu August 21, 2025 08:07 AM

You can certainly schedule integration objects, or anything else in TRIRIGA. But there's not a native method sitting there ready for this use. You could leverage the existing job scheduler, modify it to add a new type of job item, then modify the job item workflows to retrieve a specific integration object and process it. (Shooting from the hip a bit; I don't use the ootb job scheduler). A cleaner option might be to create an integration scheduler that is specifically for integration objects. You'd use the scheduler agent and its related records to do the time based triggering. You'd probably want to have integration object items so that you could tie multiple IO records to a single scheduler. You could order them to run in a specific order and only trigger the next one once all the records on the previous one are processed, etc. All sorts of possibilities. But if you're looking for a simple way to run a small number of IO processes, a few small tweaks to the ootb job scheduler should get you there with minimal fuss.

Mark Johnson's profile image
Mark Johnson IBM Champion posted Thu August 21, 2025 08:30 AM

Hey Ogi,

I'd recommend against using Data Integrator as its very limited and you don't have much control with it. Integration Object is the better approach. I ran in to this same issue many many years ago and I even have an article on LinkedIn about a surprise that I found with Integration Object. I had customized the Integration Object BO/Form in order to allow creating of scheduled events, and then the scheduled events would execute the job. However, triIntegration is considered a platform maintained object so the platform will recognize the change, and OM the OOTB versions of the object on startup. You could build a separate custom object and use scheduled events with it. Then when it fires you could have it execute your integration object job.  That's all very high level. So, I'd recommend you look in the app builder guide and read up on scheduled events.

--Mark

HIROYA OGIHARA's profile image
HIROYA OGIHARA posted Fri August 22, 2025 12:40 AM

@Mark Johnson

Thank you, Mark.
 
>Integration Object is the better approach.
I agree with you.
 
It seems your approach would be difficult for me to implement. Anyway, I'll look in the app builder guide and read up on scheduled events.
 
On the other hand, I studied Integration Object ( https://www.ibm.com/docs/en/tap/5.0.0?topic=object-integration-execution ) and come up with the following approach using TRIRIGA REST API. What do you think of my approach?
Export case
1.Define Integration Object record (named "ExportToFlatFile" here) to export BO data to a tab separated flat file. 
2.Make a shell script that has the following processes:
    1.Call TRIRIGA REST API to execute Integration Object record "ExportToFlatFile".
    2.Get the exported flat file from TRIRIGA server by File Transfer command like SFTP.
3.Call the shell script made in #2 by the scheduling tool (e.g. crontab in case of Linux).    
Import case
1.Define Integration Object record (named "ImportFromFlatFile" here) to import BO data from a tab separated flat file. 
2.Make a shell script that has the following processes:
    1.Put a flat file for import to TRIRIGA server by File Transfer command like SFTP.
    1.Call TRIRIGA REST API to execute Integration Object record "ImportFromFlatFile".
3.Call the shell script made in #2 by the scheduling tool (e.g. crontab in case of Linux).    
(I assume the shell script runs on a different server from TRIRIGA.) 
 
Regards, Ogi
Mark Johnson's profile image
Mark Johnson IBM Champion posted Fri August 22, 2025 08:48 AM

I forgot to mention, that an integration object job can be kicked off via a url. So you can actually have the scheduling outside of TRIRIGA in Ctrl-M or any other type of scheduling system.

HIROYA OGIHARA's profile image
HIROYA OGIHARA posted Sun August 24, 2025 09:44 PM

@Mark Johnson

Thank you, Mark.
 
The Documentation of Integration Object REST API (https://www.ibm.com/docs/en/tap/5.0.0?topic=execution-from-urls) says "Triggering the Execute action on an integration object record is an asynchronous event". So the REST API may return to the client before the requested import/export process completes. Do you know if we can check the requested import/export process completes by the TRIRIGA REST API?
 
 
Regards, Ogi
HIROYA OGIHARA's profile image
HIROYA OGIHARA posted Mon August 25, 2025 09:20 PM

@Bradley Stockdale

Thank you.
 
I tried to find the document describing how Job Scheduler should be modified so to schedule integration objects. I found "Creating or modifying job schedulers" page ( https://www.ibm.com/docs/en/tap/5.0.0?topic=transforms-creating-modifying-job-schedulers ), however it mentions just how to schedule ETL jobs.
I'd appreciate it if you give me a link for any documents describing how Job Scheduler can be modified to schedule non-ETL jobs (integration objects).
I've started working on TRIRIGA since last month, so I'm not so familiar with TRIRIGA.  Any information about TRIRIGA you provide for me is very helpful.
 
Regards, Ogi   

Related Content