Originally posted by: SystemAdmin

I am having difficulty getting the retry settings to work on the PC. I am running WinXP Pro SP2 with Datastage TX 7.5.1 and need the retry settings to work in the Windows environment. I have tested this from unix and they work. However, no matter what I do on the PC, I cannot get the retry settings to work.

I have a simple map with a single input card with a file as its source. I have retry on for 12 attempts with an interval of 5 seconds. It comes back immediately with Source not available. When I deploy it to unix and run it manually from the command line, it works perfectly. I have also tried it with the Windows command server and still get the same behavior.

I haven't seen any notification in the documentation, but is there something that says "Thou shalt not use Retry in the Windows environment"?

Thanks,
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

As best I know, on Windows 'retry' on a File Source will only apply if the file exists but can not be accessed (ie. locked by an FTP transfer or another application that is copying/creating the file). If the file does not exist on the first try "Source not available" is immediately returned - no 'waiting around'. I can confirm this behaviour on 6.7.1 - can't vouch for 7.5.1.

-Jim.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Hi John,

Jim is correct and the fact that it retries on Unix is a suprise! The intention for all platforms was that if the file doesn't exist, why bother retrying -- if it's just locked by some process, then retry.

-Lew
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

I went back to the documentation and also checked the help file and it doesn't seem to me that it is trying to say that it will retry a source if it exists but cannot be accessed.

From the Map Designer Reference Guide:

quote:76f33853fdThe Retry settings specify the number of times and the interval at which to attempt access an unavailable source or target.[/quote:76f33853fd]

I would believe you had I not been able to produce the desired behavior on unix. That is, I had a map running checking for a file until the max number of attemps was reached then it stopped with a source not available message. Then I ran it again and after a while put the file out there and the map picked it up on the next retry and ran successfully. Maybe this is the correct behavior on Windows (why does that not surprise me), but there was no mention of it in any of the documentation that I can find.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

John:

I don't believe you are setting up your system correctly for what you are trying to do. If you are trying to poll a particular folder and have your map execute when a file appears, you should be using the Resource Registry for that.

In the system for the input file, you should type something like:
%input_folder%*
rather than point to a directory.

Then in the Resource Registry, you need to add a Resource named "input_folder" and then specify the path.

The Configuration file will then need to be updated with the system name and the location of the Resource.mrn file.

Your system should still contain a Retry to make sure that files do get picked up after a file is unlocked by the file moving process.

Let me know if any of this doesn't make sense.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Asuming your running under the event server, you could set the input file as a source event, then the map wouldn't trigger until the file is created. However, this in itself can be a problem as if the file is ftp'd then the map can trigger before all the data has arrived. The usual method in this case is either to rename the file at the end of the ftp or sent an empty file at the end to initiate the source event triger.
If the file is being created by something like Excel, then it can be set as a source event but the retry would need to be used as Excel keeps the file locked until it is closed. This could also be the case with other apps.
You can't expect a map to retry if a file doesn't exist under Windows and anyway, why would you want to initiate the map if the file didn't exist.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Got this from Support:
quote:0b4341775fJohn,

You are correct the retry does not work when the source file is not available and will fail immediately on windows. I got confirmation from our engineer that this is working as intended on windows, but seems to be doing the re-tries on Unix which might not be the right behavior. Depending on what you were trying to achieve, you might have to do an alternative way of achieving it.

Please let me know if there are any questions.

Regards,

Arun Ramamurthy
Advanced Technical Support
[/quote:0b4341775f]

This bugs me a lot. If this is the expected behavior on Windows, then the Unix version has a bug. If they are both doing what they were designed to do (ie Windows version is known to not do retries on a source that doesn't exist, while Unix version will), then their documentation should make that perfectly clear. Even then, my understanding of the documentation, and others I work with, is that it b:0b4341775fu:0b4341775fsays[/u:0b4341775f][/b:0b4341775f] it will retry a source that is i:0b4341775funavailable[/i:0b4341775f]. Unavailable to me can mean both that the source exists but cannot be accessed currently or does not exist currently.

Jennifer, don't take this wrong, but IMHO, the Resource Registry has nothing to do with this. It matters not that I am using a source identified using a Resource Registry Alias, it still comes back immediately with Source Not Available (yes, I tested it).

Jan, the problem I have is the map that has the retry is in the middle of a string of maps and I am also using HANDLEIN to pass in a decent number of sources and really do not wish to incur the overhead of writing them out, reading them back in & re-validating them again.

I'm not trying to be difficult, it just doesn't seem like the behavior I'm seeing on windows is correct given all the documentation I have read. The Unix behavior seems to me to be completely in line with the docs.

I guess I'll have to keep digging. Thank you all for your suggestions and keep them coming if you find out anything more. I'll pass along any more information I get on it as well.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

John: I am surprised by Ascential's response on this. As you quoted directly from the documentation, retry is invoked during an attempt to access an "unavailable source or target". According to the Merriam Webster online dictionary, "available" means "present or ready for immediate use". So it follows that unavailable must mean "not present or not available for immediate use". So the unix behavior when a file does not exist sounds absolutely consistent with the documentation--retries are attempted until either the file becomes AVAILABLE (i.e, becomes present and ready for use), or until retry attempts are exhausted. If the "bug" is in unix and not in windows, both the unix code and the documentation needs to be fixed. If they "fix" it in this manner, then the capability of the product is diminished.

One of your correspondents asks why would someone want to retry on a file that didn't exist? I personally have been conditioned by the Eventserver. It will immediately pounce upon an incoming FTP'd or copied file--long before the FTP or copy completes. So what do we do? FTP/copy, then rename--meaning the file doesn't exist (isn't available) until the rename is done.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

quote:5b1505c464One of your correspondents asks why would someone want to retry on a file that didn't exist? I personally have been conditioned by the Eventserver. It will immediately pounce upon an incoming FTP'd or copied file--long before the FTP or copy completes. So what do we do? FTP/copy, then rename--meaning the file doesn't exist (isn't available) until the rename is done.
[/quote:5b1505c464]

You have missed the point. The map triggers on the name of the file so the trigger doesn't happen until the file is renamed to the desired value. It doesn't trigger on the non existence of the file.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

First off, let me explain a little bit more about what I am trying to do. I have developed a system of maps that are triggered off a single map with a watch on the Unix server. We do all our development on Windows, running our maps from the Map Designer. For development/testing purposes, I have enhanced them with an additional step of producing analysis data that will compare the TX output with the output of the Unix system TX is replacing. When a flag is set, this system of maps will go an extra step and run the Analysis Maps I have set up. Since this is for development only it runs in Windows, and only on command--not tied to an event server.

The existing Unix process is such that it takes a minimum of around 30 seconds to run. And the daemon/cron that is watching for input files for it is on a 60 second cycle. So best case scenario, the existing unix process will take at least a little over 30 seconds.

So my normal maps run and before they are finished, they see the Analysis flag is on and they run the Analysis Map. The first part produces the input data for the unix process and puts it out into a Samba share that is on the unix server. The unix process picks it up and processes it producing output in another directory. After the 1st Analysis Map put this file out, it runs another map and that map has an input card whose source is in the unix process output directory. Since the unix process takes so long, I needed this input card to retry until the output file showed up or until the number of attempts was exhausted. I had it set up for 24 attempts, one every five seconds equaling a total time of two minutes. As I have said before, it works perfectly on unix. But on Windows, it just comes back immediately with Source Not Available.

I got a call from Ascential after I posted this thread. They basically just reiterated what they did in their email. That the way it works in Windows is the way it was intended to work. They way it is working in Unix is a bug. Forgive me for being skeptic, but I have a hard time believing the unix version has buggy code, that just mistakenly happens to work the way that every piece of documentation says it is supposed to work. And not only that it does exactly what you tell it to do with the retry settings. You turn retry off, it doesn't retry. You turn it on with 24 attempts, one every five seconds and it will retry for two minutes or until the file shows up. Man! I wish I could write buggy code like that! Usually, it is the other way around, the code doesn't do what the documentation, and common sense, says it should be doing. Wait a minute, wait a minute. Isn't that what it is doing...on Windows?

The funny thing is I just created a map that has an input card, with retry on, that connects to a database an performs a query. For sake of example and testing, the query is such that it will not return a row. When I run it, it runs correctly for the set period of time specified in the retry settings. When I change the name of the directory that houses the MDQ files and then run the map, it miraculously runs the same as before, exhibiting correct retry behavior. Thinking that maybe the database connection information is compiled into the map such that my change won't matter, I correct the directory name for the MDQ files and disconnect my computer from the network, thus disconnecting me from connecting to the database. Shock of all shocks--it runs as expected, retrying until the attempts are exhausted.

So as far as I know, b:a976407a9ei:a976407a9eunavailable[/i:a976407a9e][/b:a976407a9e] means what you and I think it means on all the adapters on Unix, and all the adapters on Windows b:a976407a9eu:a976407a9eexcept[/u:a976407a9e][/b:a976407a9e] the file adapter on Windows.

I know how this works. I have been in the software business for over 10 years, nine of them with software companies producing software for customers. I know that doesn't make me an expert or some kind of guru, so don't harp on that. I started in Support and I've pulled this same argument on customers myself. "It's working the way it was designed. Have a nice day." Technically all software works the way it was designed. I've yet to see a bug that wasn't the fault of the person(s) that designed it.

Let me also clarify that Arun and the other Ascential person I spoke with were very cordial and nice. They did not try and cut me off and strove to understand my situation. Another consultant emailed a coworker of mine and said we sounded like we were upset. We are not mad or upset. I am just trying to get to the bottom of this and find out why it isn't working. So far I am skeptical of the answers I've been given. IMO, it just doesn't make sense the Unix version works the way the docs say it should and even the way the other adapters work on Windows. Thankfully, I've been able to come up with a work around that is performing adequately right now. But I would rather retry work the way I understand the documentation says it should.

John
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

John,
What's HANDLEIN? Do you mean ECHOIN.
Jan
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

No, I mean HANDLEIN. Read your docs or look at TX Help.

John
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

HANDLEIN() is a function - introduced in 7.5 - that can be used in place of ECHOIN(). It is beneficial when you wish to pass a large amount of data from one map to another via the RUN function.

The ECHOIN() function (as with -IEx) creates a copy of the data passed with the RUN command, hence impacting performance (time/workload/memory etc).

HANDLEIN() instead "points" the map that is run to the data already stored in memory, hence circumventing the need to duplicate it. The most obvious benefit is with large sizes of data.

FYI large data is defined as > 100K but your hardware would probably determine the best cut-off point.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Hi Jan
HANDLEIN is a new feature in 7.5. which works much like a functional map with run maps HANDLEIN passes a pointer to the original data instead of passing the actual data to the run map in memory. Much more efficient then
Having the same data in memory 2 or 3 times depending on how many levels deep you go in run maps.
Example map1 runs map2 runs map3
Thanks
Gary Lilley
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Ah I haven't done any new development yet in 7.5.
Your best solution is to create an empty file at the end of your unix process and use that as a trigger, then you know that the data file is complete.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Jan, don't take this the wrong way, but you continue to miss the point.

The point is b:67eeb8a8cai:67eeb8a8cau:67eeb8a8caNOT[/b:67eeb8a8ca][/i:67eeb8a8ca][/u:67eeb8a8ca] that the unix process is putting out a file that my map is triggering on before the unix process is finished with it. The point i:67eeb8a8cab:67eeb8a8cau:67eeb8a8caIS[/u:67eeb8a8ca][/b:67eeb8a8ca][/i:67eeb8a8ca] that the unix process takes a while and I wanted the Analysis process to be a one step deal. I didn't want to write out all my data (potentially a large amount of data & multiple targets), put the unix trigger file out there. Wait, check the directory, wait, check the directory, (repeat). Then when the unix process put out it's files, Execute another map that read in all the data (potentially large amounts from multiple sources), revalidate the inputs and then continue processing. The fact that the unix process is taking a while and the output from it is not ready when I want to call the map that needs the unix output. u:67eeb8a8caHence[/u:67eeb8a8ca], I wanted retry to work the way it says it will.

If you had read my detailed explanation about what I am doing, you should have been able to understand the bug (IMO) that datastage is exhibiting. The "best solution" would be for TX to work the way the docs say it will.

John
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

i:a70ec5ffa5EDIT: This post was in response to a now deleted post by someone whom I will not name that mentioned I should use an Event Server watch. They also mentioned that I might try the -Y option on the map.[/i:a70ec5ffa5]

The -Y retry option does not pertain to an Input Card's source. That retry option applies to files the map might need that are not associated with Input/Output Cards, such as trace files, audit files, etc. If you wanted to specify an override for an Input Card's source you would do so in the following manner for a card using the file adapter:
code:1:a70ec5ffa5-IF3R24:5 C:\dir\dir\filename.txt[/code:1:a70ec5ffa5]
Yes, I admit we could set up a watch for the file from the unix process to trigger the rest of the analysis process. However, in our situation, that would mean it would have to be deployed to our Unix box and this map is just for testing and analysis and I don't want to do that. We do not have a Windows Event Server license, so I can't do that in Windows. Furthermore, even if we did have an Event Server license, I don't want to put the files I am using i:a70ec5ffa5HANDLEIN[/i:a70ec5ffa5] to pass on to the next map. Requiring me to read them back in to the next map and revalidate them.

I am a developer on this project and know I will not be around forever. I wanted it to be a one step process so the person(s) that follow me don't have to do multiple steps when I know how I could make it a much more professional product that I am delivering to my customers.

John
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

I deleted it because I realised that the -Y option was not the type of Restart you were using and I didn't suggest you should use it.
Don't get so touchy.
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Oh, I'm sorry. I must have missed the point again. ;·)

Give me a break, you did suggest it. Otherwise, why would you have posted it?

And I am not touchy about my post or the whole thread for that matter. I just refuse to accept the canned support answer I was given. None of their reasoning holds water IMO. So far every other adapter I have tried, in Windows, retries u:d11586f5fdeven[/u:d11586f5fd] if the the source is not available--either because I gave it incorrect source information or disconnected my computer from the network. So I can't understand how they can say it is working as intended and the Unix version is buggy working as the incorrect documentation says it should.

John
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

quote:c184d87819Oh, I'm sorry. I must have missed the point again. ;·)[/quote:c184d87819]
Yes. Try reading it again.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

I can't read it again...you deleted it.

i:71e91fb914Tag, you're it.[/i:71e91fb914]

lol
John
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Missed the point again.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Dang! Then just enlighten all of us then.

lol
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender