IBM Workload Automation & Workload Scheduler

In my experience working with Workload Automation, I have come across many powerful features, but one of the most flexible and convenient is the file dependency functionality. File dependencies help ensure that jobs and job streams execute only when required files are available, preventing scheduling failures and optimizing workflows.

There are four different ways Workload Automation handles file dependencies:

1. File Dependencies in Scheduling
2. Event File Monitoring
3. File Monitor - Create a Job
4. Start Condition - File Monitor

Each of these methods has its strengths and should be used based on the specific requirements of your workload.

1. File Dependencies in Scheduling

The simplest way to handle file dependencies is through scheduling. This method triggers a job or job stream when a specified file arrives and holds up scheduling until the file is available.

Advantages:

• Since it is not a job, it does not count against job-based billing models.
• Easily readable and monitorable in the Dynamic Workload Console (DWC).

When to Use:

• When you expect a file to arrive on specific days.
• For dummy transactions, where an external process (unknown to Workload Automation) sends a file upon completion to trigger actions.

Example:

SCHEDULE EU-HWS-LNX275_1#SSC-JOBSTREAM01

AT 0800 UNTIL 0830 ONUNTIL CANC

:

EU-HWS-LNX275_1#SSC-DEMO-JOB01

 AT 0800

OPENS EU-HWS-LNX275_1#"/tmp/leon*.txt" (-f %p)

EU-HWS-LNX275_1#SSC-DEMO-JOB02

 FOLLOWS SSC-DEMO-JOB01

END

2. Event File Monitoring

Event file monitoring allows you to create a file monitoring event that triggers a job or job stream when a file arrives.

Advantages:

• Keeps the current plan clean by avoiding unnecessary jobs or job streams.
• Provides more flexibility for post-arrival actions.
• Ideal for files that appear multiple times in a day.

When to Use:

• When the exact arrival day of the file is unpredictable.
• When files arrive multiple times a day at inconsistent times.

Example:

<eventRule name="FILE-DEMO" ruleType="filter" isDraft="yes">

    <eventCondition name="fileCrtEvt1" eventProvider="FileMonitor" eventType="FileCreated">

        <scope>

            /TMP/LEON*.TXT ON EU-HWS-LNX275_1

        </scope>

        <filteringPredicate>

            <attributeFilter name="FileName" operator="eq">

                <value>/tmp/leon*.txt</value>

            </attributeFilter>

            <attributeFilter name="SampleInterval" operator="eq">

                <value>60</value>

            </attributeFilter>

        </filteringPredicate>

    </eventCondition>

    <action actionProvider="MailSender" actionType="SendMail">

        <parameter name="To">

            <value>leon.dodenbrett@hcl.software</value>

        </parameter>

        <parameter name="Subject">

            <value>The file %{fileCrtEvt1.FileName} has arrived</value>

        </parameter>

    </action>

    <action actionProvider="TWSAction" actionType="sbj">

        <parameter name="JobDefinitionName">

            <value>AE-DEMO-JOB01</value>

        </parameter>

    </action>

</eventRule>

3. Create a Job - File Monitor

Using the filemonitor executable from the Workload Automation bin directory allows you to configure specific file monitoring parameters.

Advantages:

• Provides advanced monitoring options, such as monitoring for a specific amount of time or requiring a minimum file size.
• Enables easy job migration from other schedulers.

When to Use:

• When monitoring must run for a specific duration.
• When monitoring must start at varying times.

Example Command:

filemonitor -path /tmp -event fileCreated -scanInterval 60 -minFileSize 1024 -timeout 3600

Couple Important Supported Parameters:

• -path: Defines the directory or file pattern to monitor.
• -event fileCreated | fileModified: Specifies the type of event to monitor.
• -timeout seconds: Defines the duration of monitoring.
• -minFileSize bytes: Ensures the file reaches a minimum size before triggering.
• -recursive: Enables monitoring of subdirectories.

4. Start Condition - File Monitor

The Start Condition - File Monitor is an advanced version of event-based file monitoring, allowing for more complex conditions to trigger job streams.

Advantages:

• Automatically creates an event to monitor for files with complex dependencies.
• Helps keep the plan clean and manageable.

When to Use:

• When monitoring requires advanced conditions.
• When dealing with dynamic agents using pools.

Example:

SCHEDULE LEON-POOL#START_COND_JS

STARTCOND FILECREATED LEON-POOL#"/tmp/start.cond.*.txt" INTERVAL 60

( ALIAS FILE_CREATED RERUN OUTFILE "/tmp/start.cond.out.txt" )

:

LEON-POOL#JOB1

END

Choosing the Right File Dependency Method

By understanding the strengths of each method, you can optimize Workload Automation to better suit your business needs. Have any tips or experiences with file dependencies? Share them in the comments!