IBM Sterling Transformation Extender

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP

Triggering will happen if the file timestamp is updated.  Both outputs are set to "sink", which suggests you are doing named adapter rules in the output cards.  If those named adapter rules change the input file in place (not a copy) that will update the timestamp and trigger another watch event.

If "multithreading setting" means Cooperative Listener mode, that should not cause a problem like this.  We have been running tons of files through coop Listener without any collisions.

Jim

------------------------------
Jim Davee

Original Message:
Sent: Thu May 02, 2024 04:42 AM
From: NeverXP YANG
Subject: How to prevent processing the same file twice?

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP

Thanks Jim.

We did not update the input file in WTX Map, the logic is that whenever the Map is triggered, it will fetch the data from the database and generate a file with a unique file name. What happened is that WTX generated two files because of this issue. Understand that WTX wasn't supposed to be like that. But it happened.

I've been using WTX for over ten years and this is the first time I've encountered this problem. Is this something to do with this setting in the launcher?

And this?

------------------------------
NeverXP YANG

Original Message:
Sent: Fri May 03, 2024 02:50 PM
From: Jim Davee
Subject: How to prevent processing the same file twice?

Triggering will happen if the file timestamp is updated.  Both outputs are set to "sink", which suggests you are doing named adapter rules in the output cards.  If those named adapter rules change the input file in place (not a copy) that will update the timestamp and trigger another watch event.

If "multithreading setting" means Cooperative Listener mode, that should not cause a problem like this.  We have been running tons of files through coop Listener without any collisions.

Jim

------------------------------
Jim Davee

Original Message:
Sent: Thu May 02, 2024 04:42 AM
From: NeverXP YANG
Subject: How to prevent processing the same file twice?

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP

We normally set our File triggers to Rollback instead of Commit for "OnFailure".  I'm wondering if that Commit is upping the timestamp on the file and causing the 2nd trigger?

We had a similar problem of double triggers a few years ago when we moved to RHEL.  ITX support helped debug it by setting all the Launcher logs to '1' and we saw the double hits with the 2nd one failing or being in collision with the first.

The problem we uncovered was letter case!  We had files with longer file names of 20-50 chars where the source was doing a alpha-numeric increment.  The alpha part was mixed case and it could write out fully identical file names except for one single alpha that changed letter case. 

Launcher was not able to distinguish that letter case change properly on RHEL and was treating both files as the same.  The OS and file system didn't care and knew they were two different files, but Launcher triggering would trip up on it.

In our case we solved the problem by having the source use upper case only for that increment.  Never had the problem again after that.

I'm wondering if you have something similar.

Jim

Thanks Jim.

We did not update the input file in WTX Map, the logic is that whenever the Map is triggered, it will fetch the data from the database and generate a file with a unique file name. What happened is that WTX generated two files because of this issue. Understand that WTX wasn't supposed to be like that. But it happened.

I've been using WTX for over ten years and this is the first time I've encountered this problem. Is this something to do with this setting in the launcher?

And this?

------------------------------
NeverXP YANG

Original Message:
Sent: Fri May 03, 2024 02:50 PM
From: Jim Davee
Subject: How to prevent processing the same file twice?

Triggering will happen if the file timestamp is updated.  Both outputs are set to "sink", which suggests you are doing named adapter rules in the output cards.  If those named adapter rules change the input file in place (not a copy) that will update the timestamp and trigger another watch event.

If "multithreading setting" means Cooperative Listener mode, that should not cause a problem like this.  We have been running tons of files through coop Listener without any collisions.

Jim

------------------------------
Jim Davee

Original Message:
Sent: Thu May 02, 2024 04:42 AM
From: NeverXP YANG
Subject: How to prevent processing the same file twice?

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP

Triggering will happen if the file timestamp is updated.  Both outputs are set to "sink", which suggests you are doing named adapter rules in the output cards.  If those named adapter rules change the input file in place (not a copy) that will update the timestamp and trigger another watch event.

If "multithreading setting" means Cooperative Listener mode, that should not cause a problem like this.  We have been running tons of files through coop Listener without any collisions.

Jim

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP

Is this a network drive?  I've seen things like this when trying to set up cooperative browsing on a SAN.  Check with your team to see if the drive is throwing NFS errors.   

Rich

Triggering will happen if the file timestamp is updated.  Both outputs are set to "sink", which suggests you are doing named adapter rules in the output cards.  If those named adapter rules change the input file in place (not a copy) that will update the timestamp and trigger another watch event.

If "multithreading setting" means Cooperative Listener mode, that should not cause a problem like this.  We have been running tons of files through coop Listener without any collisions.

Jim

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP

Hi, can anyone help me with this situation?

We placed a file to a WTX map, but the file was processed twice. Even the second time we got a source not available error.

Is it because of the multithreading setting?

Settings for Input card: Onsuccess=Delete, OnFailure=Commit.

------------------------------
NeverXP