Originally posted by: IBM WTX

One of my client using IBM WTX 7.5 in cluster envirnoment, with windows 2003 server, 8 GB RAM. Last year I implemented a patch to fix this problem. Now this problem reoccured. When they try to process large file for transformation e.g. flat file holding records 200,000 or 300,000 to XML file. WTX gives error not enough memory to execute maps. Does any one else face this problem of memory leakage? What are the best possible ways to protect memory leakge. I will appreciate your comments.

I am able to process a 10MB flat file into XML easily. But if the file size goes over 20MB having more then half million records. It gives me the above error "Not enough memory to execute map"

I am using the following setting for my router map.
Map delay: 500
Max Concurrent Map Instance: 1
work space: File

Calling MAP setting
Map Delay: 500
Max Concurrent Map Instance: 1
Workspace: File

In my router map I have the following setting for

WorkSpace
PageSize 64
PageCount 8
Directory Map
WorkFilePrefix Unique

Fetch As: Integral
Work Area: Create
Fetch Unit: S

For the actual map which is doing transformation I have this setting
WorkSpace
PageSize 64
PageCount 8
Directory Map
WorkFilePrefix Unique

Fetch As: Integral
Work Area: Create
Fetch Unit: S

Any idea why this problem is happening, or this is a limitation or bug from WTX? Eeven I tried the same maps in IBM WTX 8.1 the same problem. I will appreciate your help in this regard.

Regards
Umer Farooq
+966 507432082
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: TimUK

there is no restriction in the product in the way you'v explained it.
I'd start with the page size and count settings the ones you've got are the defaults.
Its not an exact science and depends on the input data complexity and the number of items in the tree and then how many output cards you are using, how many times you reference the inpout data.
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: TarunB

I'm surprised that the map is bombing out at just 20Mb, but I guess that's Windows for you. The problem is most likely related to the building of the XML file, seeing as XML is very heavy on the metadata front.

Have you tried using Burst mode on the input flat file?

Alternatively, you could use a 'pre-processing' map to split the file into smaller chunks and then process each part individually, maybe appending all files in a 'post-processing' map if required.

If you're using a RUN command, try to re-code this out of the flow because large amounts of data will be held in memory. Or ensure that HANDLEIN is being used instead of ECHOIN or -IE etc. Perhaps use triggers instead of RUN for the called map.

We process files over 100Mb in size regularly and I have personally tested a 250Mb file here as part of a complex MQ process (splitting/joining), so certainly the limitation you face is not a TX one.

Good luck - let us know how you get on.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: IBM WTX

Hi TarunB,

Thanks for your reply. For the burst mode I need to change my design, which I dont want to do at this stage. Beside this customer has one Outbound flat file which is being transalted to XML and then uploaded to Centeral Repository. In other words one flat file input and one xml output. If I change the design to burst mode or some pre processed map then there will be multiple output coz I am using PUT in my final card. Which customer dont want or I have to consolidats these burst or pre processed small files outputs into a single output file.
After making some changes for Page size 512 and Page count 16, I tried to process around 18MB flat file with 3,30,000 records successfully. But large file over that still giving me problem. I have the following setting in my Router Map

=RUN( "S:\SADAD_LINK\MAPS\" + MapName Field:.:FileInfo , "-IE1S" + NUMBERTOTEXT(SIZE(FileName Field:.:FileInfo)) + " " + FileName Field:.:FileInfo
+ " -IE2S"+NUMBERTOTEXT(SIZE(PACKAGE(InFile) )) +" " + PACKAGE(InFile)
+ " -IE3S"+NUMBERTOTEXT(SIZE(PACKAGE( CfgRec Record RouterCfg:.:FileInfo) )) +" " + PACKAGE(CfgRec Record RouterCfg:.:FileInfo)
+" -OF1 S:\SADAD_LINK\ERR\" + FileName Field:.:FileInfo+ "." + DATETOTEXT(CURRENTDATETIME("{CCYYMMDD}{HH12MMSS3-3AM/PM}" ) ) + ".ERR" )

Over here I will try Handle In as you sugessted and see the results. My called map whis is translating the flat file into XML has the following argements for Out put sink map.

=IF(PRESENT(XML) ,
PUT("FILE",
OutDir Field:Route+ SUBSTITUTE(TextOnly Other:TextOnly File:FileName, "CA","CAXML.BUPLRQ")
+"."DATETOTEXT(CURRENTDATETIME("{CCYYMMDD}{HH12MMSS3-3AM/PM}" ) )".Xml"
,PACKAGE( XML)))

If you have any comment on above argument I will apreciate, mean while I will try handle in and get back to you. Thanks for your vlaued input.

Regards
Umer
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: DianeC

Are there any restrictions on the server itself that may be causing this error? Limits on the size according to user id? (I ran into that one once on AIX) Limits on the amount of total space allowed for WTX? or the directories the processing is taking place in? i agree that it doesn't seem like a WTX issue. . .
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: IBM WTX

Hi Dian,

The current envirnoment is MS Windows 2003 with SP1 and 8GB RAM. Which Ifeel enough for the current transformation. Still working hard to get rid of that problem. Trying TarunB idea to change IE with HandleIn. If I change my code to handle in my router map is not working properly. It doesn't delete the file and keep processing the same file. Presently looking into this issue with handly in. The rest I tried almost everything like Page Size and Page countr setting, even StreamMaxMemlimit and StreamMaxDirectory parameter as well. But no luck so far.

Regards
Umer
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: mikeladd

I think TarunB hit the nail on the head. Increase your Page Size.

It isn't an exact science, but I can tell you that 8 pages at 64k each, will most likely cause problems with a 20 meg XML file.

I would bump up the RUN map (unless the Router is also parsing the data) up to maybe 16 or 24 pages with a page size of anything from 128 -> 256. Once you find the setting that works, bump it up a little more in case you get an even larger file down the road.

It is trial and error, but if you can reproduce it locally, then it doesn't take that long to run the file through to make sure it works.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: mikeladd

Sorry, that was supposed to be TimUK who mentioned the page size.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: IBM WTX

Hi Mike,

The flat file size over 20MB which can contain around half a milion or more reocrds causing problem. I chenaged the page size to 512 and page count to 16, but no fruit. This setting was sugessted by dtxpage utility. Any ideal setting when you knew that the output will not exceed over 2gb.

Regards
Umer
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: phil2030

Which map gave you the error, the "router" or the "transformation" map you are running?

If it is being returned by the "transformation" map, what happens if you run the map directly with the same data? Does it work or return the same error?
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: IBM WTX

Hi Phil,

The transformation map gives me out of memory error, I have observed when the meta data files goes over 600MB I got this error. Workspace set to file Pagesize 512, Page count 16. Server has 8GB of RAM. So far I was trying with router map. When the file size grows over 20MB in flat file which contains around half a million records, in that case I get this error. The point is why WTX not realease the memory when it reaches to max peak memory? Even I tried the setting in DSTX.INI , StreamMamMemLimit = 250 and StreamMaxMemDirectory parameters. No success so far.

Regards
Umer
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: DianeC

Are you 32 or 64 bit? You say you have 8GB of RAM, but is that dedicated to WTX? Is there anything else running on the machine that would possibly be taking up the memory/cache? In the past I've heard that some programs were memory hogs on Windows.
After reading down through the other posts, I'm still not convinced at this point that it is a mapping issue. As TimUK & TarunB state, you shouldn't be having an issue with processing files of that size through WTX. I think it's more an environment issue. And, I'm a non-Windows person, so don't beat me up too badly if I end up being way off base.
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: phil2030

I think as DianeC has stated, unless your running a 64bit OS and a 64bit version of WTX on said OS, the 8GB is fairly moot. You're limited to 2GB max if memory serves. Having said that, 2GB should be more than enough. When you say; "I have observed when the meta data files goes over 600MB I got this error", are your referring to the workfiles?
I will assume there are no quotas or diskspace issues...

Have you created a test map with one input card (the tree you are using) and no outputs and ran that against the data? Does it work? If you haven't done this, then I suggest you do it now. It will show whether the issue is associated with the input tree or the subsequent output generation.

Does the business map work if presented with the data directly? (E.g. from within the map designer)

Have you used process explorer to monitor how much memory is consumed for the execution engine process?
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: JulienT

I already encounter problems of "Not enough memory to execute map" by using sink card that need more than 800 Mo to store data. I think there is a max size of data that can be put in memory per card.
So you should store your data in a temporary file then use it for your run instead of putting the data directly with ECHOIN or HANDLEIN.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange