Originally posted by: SystemAdmin

Hi,

Iam using Database adapter for 2 input cards.Iam trying to put the data into flat file.
While running the map,iam getting "Not enough memory to execute map"
I have nearly 1.5L + records in each input card.
I have made settings of PageSize to 1024,PageCOunt to 1000 and used Burst mode with fetch
Unit as "S".
If i specify FetchUnit to 15,000 ,iam getting correct data upto 15000th record,remaining iam
getting duplicate data.
Is there any solution,can anybody help me in this regard.

Thanks & Regards
Vani
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Hi,

Try with the default value for PageSize (64) and PageCount (8).
PageSize and PageCount are use for memory optimisation.

Vincent
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: TarunB

Yes and TX will automatically increase the Page values as it needs. Also ensure write your workspace to a file for this map.

Are you running this on your local PC or on a server?

With Burst, there is no point setting the Fetch as S. Try using a lower value such as 1000 and work your way upwards to an optimal performance.

Perhaps I am being naive but what do you mean by "1.5L records"?
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Iam running on my local PC.
I have already made settings for pagesize(64/1024) and PageCount(8/1000) with Burst mode Fetch Unit as "S",even iam getting "Not Enough Memory to Execute Map".
I want to retrieve records from 2nd input card(2nd mdb) which are not present in 1st input card(1st mdb).
With Fetch Unit as nnnn,iam getting correct output upto that record(nnnn),remaining iam getting unwanted records.
Nearly i have 1,50,000 records in each input file.

I used the below map rule :
=f_eligibilty(EXTRACT(Row:Eligibility_Extract1 WHERE NOT( MEMBER( Row:Eligibility_Extract1, Row:Eligibility_Extract ) ) ) )
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: LaurentB

Hi,

be careful with burst mode when you need to associate two inputs :
If you use burst on one card only then all other cards are read in full for each burst. This may explain the duplicate results you get.
If you use burst on two cards, then you kind of "synchronize" those two cards (first burst of first card with first of second card, then second bursts together, etc...).

I partiallly disagree with Vincent ( sorry Vincent ) : of course you use paging settings to optimize run time, but you also sometimes need to set them properly just so that the map runs as you need it to run.

Therefore, if I understand your problem correctly, you need to read your cards as integral so that comparisons can work, and find the appropriate page settings are used so that you are able to run the map. Running the map with workspace file seems a good starting point, then may be the dtxpage utility could help you identifying the best paging options for faster execution (IMO, it is almost always a good idea to run a map with large data using workspace in files)

Hope it helps
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: TarunB

In addition to Laurent's several good points, would it be possible to replace your use of NOT (MEMBER) with a SEARCHUP function? Performing a cross-reference between 2 such large files is probably what is killing your execution, so this binary search will speed things up.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Finally,i got the output.Thank you Laurent and others.
I have used the default settings of PageSize & PageCount with Burst mode,fetch Unit as 100 for 2nd input card & the other input card is read in full for each burst i.e with Integral.
But,it is taking 1 hour to execute the map.By default page settings,we can speed up the execution,but it is not happening so...
Even if i increase Page Settings,it is taking more time to execute the map.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: LaurentB

It may be worth considering the burst size, so that you optimize the integral reading of other cards.

I'm afraid a "trial and error" process is needed there.

It may also be worth running the profiler on your map to try to identify bottlenecks (Tarun's suggestion of using SEARCHUP is definitely worth a try if applicable, I've seen increase as big as map being 60 times faster when using this in some cases)
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

By optimization of map rules,i am able to run the map faster.
Thank you Laurent & Tarun.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: jvanboga

I've found that when a map takes longer than I believe it should it's usually a design problem. If you are using 8.0 and above you should be able to run the map with profiler on. That should tell you where your map is spending it's time. Based on your results you MAY be able to modify the map accordingly.

Keep in mind that if you are doing a lot of lookups or searchups/downs your going to consume a lot of resources and time. I had a map that dealt with small files (one was under 10mb the other under 15mb) that required one lookup for every record in the smaller file. The map took forever to run. I resolved the problem by loading the data I needed from each file into a couple of temp tables. Then writing a query which extracted the data I wanted grouped together in the fashion better suited for my needs. The job went from running longer than 60 minutes to under 7 and the maps still did all the work (from temp table truncation to final output), and still do. It also resolved some issues intrinsic to lookups and searches.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange