Originally posted by: DianeC

We're running Datastage TX 7.5 & would like to know if there is a size limit on an input file and if so what it is. Does anyone know what it is or where I can find that information? I tired searching this site & didn't get a hit.
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Hello! The default for input files is a 2 GB limit.

This can be increased to 6 GB with a software patch.

The patch here was for Linux but assume it would be the same for Unix too. Not sure what platform you are using now.

For version 8.0 the 2 GB limit still holds and another patch for that version could also move it up to 6 GB for input files.

Call TX support to request the patch.

Hope this helps.

Thanks!

#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: DianeC

That helps alot - THANKS!!!
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: jvanboga

Once a year I get a file that's in excess of 4GB that gets converted from EBCDIC to ASCII. I've also tested this with larger files that required more logic. Haven't needed patches (not to say I don't need them, just that I haven't)

I haven't faced a file size issue yet and I've tested with some pretty large files. The problems I did have were based on map configuration and machine resources.

If you run the map and watch the file activity in the map path you'll note that resources are consumed at an exponential rate, ie a 6GB file can easily consume 12+ GB of resources.

Minimize what gets passed to functional maps, if possible use burst mode and use temp files vrs memory.

BTW: on windows running windows may be diff on other platforms
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

The patch Bob is referring to is for Linux only and is for the creating of very large files, not processing of them. The patch was simply a recompilation of the Linux binaries to include large file support.

As Jason points out, many times the limitation of the input file size is due to map design. On most OS's you have a limit of 2G per process. So if you have a 12G file, you will not be able to "blob" it through the map. Again as per Jason, you will have to use burst mode and use workfiles.

My tests have shown that I can only blob (pass through map that does no validation of data) about a 450M file through a map.

Hope this additional information helps.

Resistance is NOT futile.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: jvanboga

Not sure about unix or linux....

My tests in '02 showed aprox the same 400 - 500mb limitation in Windows. Interestingly enough.... changes in map design allowed a 700mb file to process faster than a 350mb file. Same file format, same data, less overhead....

John
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Sorry John, thought you are were a Jason!

Resistance is NOT futile.
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: jvanboga

:)
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: DianeC

I'm not sure if this is the right forum, but the file keeps giving me a SIGSEGV (which in our experience usually is a memory issue) when trying to process it through Commerce Manager (7.6). Here's the report:
-- ABORT report -
-- Program Name: launcher

-- Thread Reference: 14 -
Signal Received: SIGSEGV

Component: sortx12
Map: /asc/cm/cmgr_v7.6/mmgr/sortx12.mmc

INACTIVE Input Card 1: 0 /asc/cm/cmgr_v7.6/mmgr/sortec/edi000000520060818oi.x12
INACTIVE Input Card 2: 0 /asc/cm/cmgr_v7.6/mmgr/share/mmgr.conf
INACTIVE Input Card 3: 0 /asc/cm/cmgr_v7.6/mmgr/share/postlnks.tsv
INACTIVE Output Card 1: 113
INACTIVE Output Card 2: 113
INACTIVE Output Card 3: 0 /asc/cm/cmgr_v7.6/mmgr/audit/cop/edi000000520060818o_sortx12.cop
INACTIVE Output Card 4: 0 /asc/cm/cmgr_v7.6/mmgr/ack/edi000000520060818o_997.tmp
INACTIVE Output Card 5: 0 /asc/cm/cmgr_v7.6/mmgr/audit/x12/fa_in/edi000000520060818oi.997
INACTIVE Output Card 6: 113
INACTIVE Output Card 7: 0 /asc/cm/cmgr_v7.6/mmgr/alert/edi000000520060818oi.x12_in_rej
INACTIVE Output Card 8: 113

EXIT type: 0
Cleanup: 0

RUN map: <none>

-- end of ABORT report

I have changed the workspace from memory to file in the sortx12.mmc map (and some of the other sort maps), but keep get this error. Any thoughts??
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

I would suggest contacting support. They will likely request a UI against the core file created. From my perspective, no matter what the size of the file is, you should not core but get a different specific message about memory.

Yvonne Ricard

Resistance is NOT futile.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender