Originally posted by: SystemAdmin

I currently use DataStageTX to translate very large files (1-2GB) and expect to contend with even larger files (up to 4GB) in the future.

Experience bears out that the event trigger will not fire in DataStageTX when files are presented to a map that are larger than 2GB. I have also heard that using the burst mode offers some hope for dealing with files over 2GB.

In the test cases, I am working with large flat files comprised of multiple record types with no record terminator. I am using the ID=IDlast component rule method to define a burst instance and setting the burst count to 1. This approach has been successful in bursting with files up to 1.8 GB, but the map does not trigger when files are presented over 2Gb.

With this limitation, I'll have to go outside of the app to try and intelligently shrink my input files down to a size DSTX can handle, which gets complex.

I wonder whether there is a setting in an ini file that can disable the 2G trigger limit? anyone out successfully dealt with files larger than 2GB?
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

This is W2K, version 652 right?
jvanboga
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Setup a simple event. 1 large input file (2.68 gig) to a single output. No file modification or evaluation. Ran in roughly 8 minutes. Also ran a 5.3 gig file in about 22 minutes.

Its not a true test of what you are doing but its an indication of resource issues verses product ability.

My test machine is W2k, 1.2 Ghz, 2.5 Gig Ram.

System no Logs, no Trace, workspace file, pagesize/count 64/8, Burst mode.

jvanboga
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

I have large files with multiplre record types too. But I didn't success to use the burst method. Could you explain how you proceed ?

Thanks.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Thanks jvanboga - your tests are helpful. I am running on SunSolaris 5.9 with ES 6.7.1 so we do work in different environments, but it's good to know that you were able to process files that size in W2K.

I may try to simulate your test with a simple single-object to single-object map with no validation on the possibility that the complexity of the I/O objects comes into play. Because the map didn't show any evidence of triggering, I wonder whether map construction is a possible culprit or not.

Whether or not that works, I may also try a similar test but 'bursting' on a repetition of objects that are defined only by size=1byte to 10KB. That should be a simple enough burst definition that if burst was going to work, it would work here.

Patrick - let me know if this gets to your question - the burst method when you are using the FILE adapter relies on defining your input card type tree with at least one high-level repeating object, like a file that contains record (1:s). If you set the ES to operate in burst mode with a count of 1, it will execute the map once for each 1 record. Presumably, it releases resources claimed for processing of the first record before it starts processing the second record.

I'll let you all know the results.

Regards,

Karl
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

As faar as I know there are a limit of around 2Gb in the OS/390 (mainframe) environment that is caused by lack of 64 bit support
in standard classes supported by IBM.. might be something similar on
solaris....
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Hi,
my experience with large files is:
a) there is no problem when you have record end identifiers i.e. the large file consists out N records
b) I had big problems with a large file (> 1GB) which consisted out of one single record. As soon as I processed the same file with <CR><LF> there was no more problem

Hope this helps
Guido
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

color=blue:2154fd0a51still no breakthrough - [/color:2154fd0a51]

bursting in 10KB chunks did not allow the trigger
simple in/out map in burst mode (a-la jvanboga) did not trigger
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

THANKS ALL

I'll continue working on this and let you know if I come up with anything... So far I'm inclined to think that there's something between the app and the OS that is preventing it from triggering on files over 2G. It may be simliar to the OS/390 issue...

Regards,

Karl
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

As a workaround, you could possibly use a "trigger file" separate to the large file for the purpose of triggering... This is done on some systems when FTP is involved using large files and we want to be certain the file is complete before starting the map...
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

thanks tim. I'll give it a try.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

SUCCESS.

Using a combination of a separate smaller trigger file and bursting in chunks of 10KB, I was able to move a 2.5 billion byte file through a map. Both the smaller trigger file and the burst mode are necessary.

Tim, bear with me here as I theorize about why your advice worked... Can I infer that the code put in place to try and prevent files from triggering prematurely when still in transfer can no longer investigate the size/status of files over 2G (because of 32bit addressing) and therefore it prevents the map from being triggered? Was new code to prevent premature triggering included in 6.7.1?

Thanks to you all!

Regards,

Karl
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

I cannot confirm your inference... sorry. I would only be speculating. I don't recall reading anything "new" about delayed triggering, but either renaming a file upon ftp completion or dropping a trigger file afterwards is a long-time workaround.
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Sorry I wasn't of help!

This is a prime reason why we should all be 100% Microsoft!
jvanboga
PS: Just kidding, don't throw stones.

PPS: Unless of course you work for Microsoft in which case you can send me a check at 192..........
PPPS: Must be bored today
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange