Originally posted by: gunturi_s

Hello,

I have a input file of 90M with ^ delimited of each record, the data is unique for each record in the file and probably has a 500k records in the file, Map is written to take an different input source which has a 16 digit unique number .

Input 1: File of 90MB , 500k records ^ delimited fields

Input2, Smaller file probably 30 records , only one field which needs to be mapped with first field of input1 file.

Attaching the map and type tree and sample fies, i tried running the dtxprofiler and it clearly shows the time spent in validation card function internal to wtx. Any suggestions how to improve the mapping time.

Originally posted by: Ejay

Hello,

 If i just read the input ( no output tree provided in attachment )

This is my profiler result using ITX 9.0 on windows :

VALIDATE_CARD            0          1         92808    WBS00060_TF             
VALIDATE_CARD            0          1             3    WBS00060_TF             

Audit log

<Platform> Platform API for Windows - Version 9.0.0(172)</Platform>

<Burst count="1">

<ExecutionLog   burstreturn="0" ElapsedSec="0.9281">

<inputstatus    card="1"   bytes="45422434"   adapterreturn="0"   contentreturn="0"/>
<inputstatus    card="2"   bytes="400"   adapterreturn="0"   contentreturn="0"/>

What are your results and what timing do you expect for reading a 44 mb file like the one attached .

Ejay

Originally posted by: gunturi_s

Hi Ejay,

Thanks for replying, i had to trim the file to 40 MB as i couldnt upload full file here,  the actual file is double the size what i uploaded. I have a realtime websevrice which takes an input in request goes to mainframe via mq fetch some records then take one of the fields from the output to search against this file. Currently i achieved a 6 seconds best time on my solaris for only mapping the input of this file. I have to keep the mapping under 2 seconds. Will keeping the file in resource registry helps me any faster results? or component rules is the way to go

Originally posted by: paul.brett

Have you considered using the SERCHDOWN() function instead of using the EXTRACT() function (which is expensive in CPU terms)?

Bear in mind the input data will have to be sorted first.

Thanks.

Paul.

Follow me on Twitter     <--- go on, you know you really want to!

Originally posted by: gunturi_s

Hi Paul,

Currently i am still focusing on mapping the input itself as its taking the highest time. I will try searchdown instead of extract definitely, Below are the options i have to try

1. Writing component rules and map the file in input card
2. USE get function in output card instead of input mapping---- tried today still at the 6 seconds mark--
3. Keeping the file in resource registry instead of doing for every single webservice call.. --will this result in any improvement?
4. Cut the file into smaller chunks and lookup only that file which may match the first 5 digits of my input--- rough and raw approach as i would have to define logic in the map to lookup the exact file.

Originally posted by: Ejay

Hello ,

Keeping the file in resource registry , would mean that the file is always the same... ?

I'm not sure it's a good idea to keep a 100 Mb file in the RR ?

If the same file need to be read at each map execution the best option is to reuse the workspace for your input card ...

One important thing to consider is also the hardware and the speed of the disk where the data are read / written .

EJay

Originally posted by: paul.brett

Using your map (with the output cards removed) I see the following:

ElapsedSec="1.4093"

However, if I simplify the input structure, so that the row contains only 2 fields (MRCH_ACCT & EVERYTHINGELSE) then I get:

ElapsedSec="0.7802"

Not much of a speed improvement, but it's something you could consider.  It certainly shows that the biggest resource hog here, is not the input validation, for sure.

Thanks.

Paul.

Follow me on Twitter

Attachment(s)

WBS00059_2.mtt   8 KB 1 version

Originally posted by: gunturi

Thanks Paul, is that in 9.0? or 8.4 . Currently i have only 8.3 in my pc trying for higher versions

Originally posted by: paul.brett

8.3 version.

Thanks.

Paul.

Follow me on Twitter

Attachment(s)

WBS00059_2.mtt   8 KB 1 version

Originally posted by: gunturi

Thank you, i did made rest of fields as blob except merchant account number while mapping, but i didnt understand the point you made here .

It certainly shows that the biggest resource hog here, is not the input validation, for sure.  i did tried to make in integral mode with 50k units at a time.

Originally posted by: gunturi

***UPDATE****

We made the map not to directly map the file , instead in the output card we are trying to trigger a shell script which does an egrep on the main file to create a smaller output. And then reading that output into map . response time is still being tested and monitored. I will close this post by next week