Hi -

I am in search of a way to mass load data faster. I posted a similar topic and the reply said to use inoxmld from the command line. I am currenly doing this. I loaded an input file with approx 236,000 documents, approx 145MB. There are 2 standard indices and two text indices that are being built according to the schema and it took over 11 1/2 hours to load. Ultimately, I want to have 5-6+ million documents in a particular collection - but I cannot wait 2 weeks to get the data loaded.

Any suggestions/recommendations?

thanks,
tkilleen

Hi tkilleen,
Which kind of machine and operating system do you use?
How big is the amount of indexed text?
Do you switched the word fragment index on/off?
How big is your memory, buffer pool size, …?
Do you use TSD2 or TSD3?
…
We need more information to give you a hint.

Best regards,

Joachim

Hi Joachim -

1)Which kind of machine and operating system do you use?

I am running the load from a Solaris Fujitsu 600 with 6 600MHx processors and 12 GB of memory.

2)Do you switched the word fragment index on/off?

word fragment index is off.

3) How big is your memory, buffer pool size, …?

buffer pool size: 200 Effective Value: 200 MBytes High water: 78 Dynamic: no


4) Do you use TSD2 or TSD3?

I believe we have TSD3

5)How big is the amount of indexed text?

Not exactly sure what is meant by this question - below is an example of 1 document - the text index is built on the title node. All documents have similar structure.


25142
Felix Ramon Ramirez v. U.S., John Thompson, Deneise Dungee, Venson Davis, Sharon Dooley, James Fitzgerald, Tracey Ann McCormick, Frederick Smith, as Agents, Servants, Employees of the INS, County of Hudson, Hudson County Sheriff’s Office, Joseph T. Cassidy, as Sheriff of Hudson County, Hudson County Jail, Rob Reincke, George Kochell, Thomas Foley, Lillie Bale, Trish Gonzalez, Alfred Crawford, Carlos Carames, as Agents, Servants, Employees of County of Hudson
Ramirez v. U.S.
1072

1
4
2000

WALLS

2000
999
2624


81
4637
532




hope this info helps.

tkilleen

Hi tkilleen,
Do you run the machine with Solaris 8?

Best regards,

Joachim

Hi tkilleen,
If your are using Solaris 8 on 64 bit, you can get a hotfix from support (workaround for an I/O-Problem in Solaris 8).

I hope, this will help you.

Joachim