Hello everyone,

I have a 17 MB XML file. I used XML SPY Pro to generate an appropriate DTD for my XML file. I imported the DTD with the Schema Editor to the database and have imported the XML file to the database through the Command line. (Actually the data in DB is smaller than the XML file, it is just 10 MB)

Everything worked fine. I tried to use XQuery over interactive interface and it works fine when the result is limited to 16 results.

I use JBuilder Foundation X to create an application. I use the Tamino Java Api and could successfully query a smaller XML collection from the DB.

Now the problems:

1) If I try to query the bigger XML collection (10 MB), I get the famous INOXYE9291, Transaction aborted because it has taken too long. I have searched in the forum and have found the idea of increasing the transaction time through Tamino Manager. But where is this option exactly? I can’t find it.

2) From my experiences with other XQuery in-memory solutions; this 17 MB XML file made the Java VM crash every time with the message: java.lang.OutOfMemory. But if I had increased the Java VM Memory allocation with the parameters -Xms768m -Xmx768m in JBuilder I had no problems anymore.

The Situation with Tamino is different: I have on the one hand my JBuilder IDE, on the other the Tamino Server which allows me also to increase the Java VM memory. I have just 1 GB RAM, so who shall get the ‘-Xms768m -Xmx768m’ parameter? JBuilder or Tamino Server?

Any comments?

Thank you
Houman Khorasani
University of Wisconsin Platteville

Houman,

That ones indeed a very interesting situation. Firstly, let me tell you about where should you set the transaction duration. The transaction duration is a server property therefore you’d find that at the following location within the Tamino Manager -

“tamino > database_name > Properties > Server > maximum transaction duration”

Well, now about the VM memory settings. I think you would’nt need to use it at either ends, provided you use cursors and fetch records in multiple chunks and parse them using SAX. Tamino would’nt need it because internally tamino manages any XML content in an optimised memory efficient way.

Try that out, if you need any help we are here…

Rahul Roy

Tamino API for Java
Software AG (INDIA)

Hi Rahul,

Thanks for your reply. I have successfully modified the transaction duration to 900 s.

This time I get a different error message:

Tamino access failure (INOXME8504, XML maximum query duration exceeded)

Well, maybe I should first explain more about the data itself.
The XML file I had imported has 234964 datasets.

Each dataset has three elements.

Like this:

  
<data>
<row>
<dat>20000801000000</dat> 
<tick>17328</tick> 
<dlay>48966</dlay> 
</row>
<row>
<dat>20000801000000</dat> 
<tick>22926</tick> 
<dlay>48676</dlay> 
</row>
...
</data>
</pre><BR><BR>I would like to use this XQuery statement:<BR><pre class="ip-ubbcode-code-pre">
for $wb in doc("wisc_berkeley.xml")/data/row
where $wb/dat='20000801000000'
return
<ROW>
{ $wb/tick }
{ $wb/dlay }
</ROW>
</pre><BR>This Statement should return 80809 <ROW> tags, containing the elements tick and dlay.<BR><BR>You ere talking about cursors and how they fetch records in multiple chunks, you also said using SAX for parsing the result.<BR><BR>I wrote a method to write these 80809 resultsets to a XML file.<BR>I don't know if this is the most efficient way.  If you know a better way, I would love to learn it.<BR><BR><pre class="ip-ubbcode-code-pre">  
TConnection connection = TConnectionFactory.getInstance().newConnection(databaseURI);

TXMLObjectAccessor accessor = connection.newXMLObjectAccessor(TAccessLocation.newInstance(collection),TDOMObjectModel.getInstance());

TResponse response = accessor.xquery(xquery); 
TXMLObjectIterator iterator = response.getXMLObjectIterator();

File outputFile = new File("targetFilename.xml");
OutputStream outputStream = new FileOutputStream(outputFile);

PrintWriter writer = new PrintWriter(new OutputStreamWriter(outputStream));

writer.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
writer.println("<RESULT>");

while(iterator.hasNext()){
TXMLObject xmlObject = resultset.next();
StringWriter stringWriter = new StringWriter();
xmlObject.writeTo(stringWriter);
writer.println(stringWriter.toString());
writer.flush();
}
writer.println("</RESULT>");
writer.close();
outputStream.close();

I tried to debug it. Exactly at this line
TResponse response = accessor.xquery(xquery);
the problem occurs.

Maybe here is the spot where I need to tell him get first 10000 datasets and fetch them first and write them to the outputfile, then do the next 10000 and so on.

Is this idea right?
But how do I do that?

Thank you
Houman Khorasani
University of Wisconsin Platteville

[This message was edited by Houman Khorasani on 13 May 2004 at 23:44.]

Houman,

You just need to change the following lines in your code.

TXMLObjectAccessor accessor = connection.newXMLObjectAccessor(TAccessLocation.newInstance(COLLECTION),TSAXObjectModel.getInstance());
TResponse response = accessor.xquery(xquery,10000);

Try it out and please get back in case of any problem.

Regards

Nilesh Y.

Hello Nilesh,

Thank you very much for your reply.

The compiler had a problem with

TSAXObjectModel.getInstance()</pre><BR><BR>So after I have inserted the line<BR><pre class="ip-ubbcode-code-pre"> import com.softwareag.tamino.db.api.objectModel.sax.TSAXObjectModel;

to my class, it said then for .getInstance()
“Cannot resolve symbol method getInstance() in class com.softwareag.tamino.db.API.objectModel at line…”

The class has no method with this name. How can this be?

It’s now 3:40 in the morning, maybe I am just tired and don’t see it…

Regards
Houman Khorasani
University of Wisconsin Platteville

Houman,


I think with the following lines your code should work.

TXMLObjectAccessor accessor = connection.newXMLObjectAccessor(TAccessLocation.newInstance(COLLECTION),TDOMObjectModel.getInstance());
TResponse response = accessor.xquery(xquery,10000);

I think it’ll solve your issue. Using a SAX object model requires some more implementation. So please try out the above solution and if you still face the same problem the we’ll talk about SAX object model.

Thanks

Regards

Nilesh Y.

Hi Houman,

Well, to use SAX Object Model is a bit tedious because of the following reasons:-

The factory method getInstance() is a singleton instance of the following “generic” classes
TDOMObjectModel,TJDOMObjectModel,TStreamObjectModel.

While TSAXObjectModel is “not generic”. Here,a specific instance of TSAXObjectModel must be created with 1 or 2 sets of helper classes. This set includes eventhandler (elementDefualtHandler and documentDefaultHandler) which provides the processing logic and a class respresenting XML node either as a document or an element(use saxElementClass or saxDocumentClass).


For example:-

For processing query results u have to use saxElementClass(e.g.elDefHandler in below code) and elementDefualtHandler(e.g. YOUREXAMPLEDefaultHandler in below code).


after u do that u need To construct a TSAXObjectModel as shown below.


// Instantiate the default handler that processes the sax events
YOUREXAMPLEDefaultHandler = new YOUREXAMPLEDefaultHandler();

// Instantiate the document and element event handlers each of which
// delegates its events to the greetingDefaultHandler
elDefHandler = new ElementDefaultHandler( YOUREXAMPLEDefaultHandler );

// Instantiate the specific TSAXObjectModel
saxObjectModel = new TSAXObjectModel(“YOUREXAMPLESAXObjectModel”, YOUREXAMPLE.class, YOUREXAMPLE.class, docDefHandler, elDefHandler );

// Do the object model registration.
TXMLObjectModel.register( saxObjectModel );



//Obtain a SAX Accessor
accessor = connection.newXMLObjectAccessor(TAccessLocation.newInstance( collection ) , saxObjectModel );
}


U need to pass YOUREXAMPLE.class in which ur main method is present.

If this doesnt clear ur picture, get back,

Regards,
Sonal B
SoftwareAG India

Hello Sonal,

Thank you very much for your response. I am still working on your solution. I have read about SAX and understand the basic how it should work. I still have problems with the registring the handler or creating them in code. I need more experience in that.

Your Example wasn’t very clear to me, so I also referred to the Tamino API Java Doc → TSAXObjectModel and could figure out some more stuff. But I’m still confused.

TSAXObjectModel saxObjectModel = new TSAXObjectModel("YOUREXAMPLESAXObjectModel", null, ??? , null, elementDefaultHandler);
</pre><BR><BR>This is the line of code I have problem with.<BR><BR>Lets check the parameters together<BR><BR>1) <B> specifier </B> - <I> a string identifying this SAX object model instance. As it is possible to instantiate more than one instance, each instance must have a different specifier. </I><BR><BR>In my case it does mean this string can be anything since I have just one instance, right?<BR><BR><BR>2) <B> saxDocumentClass </B> - <I> a class implementing the TSAXDocument interface. This parameter is required if the documentDefaultHandler parameter is not null, otherwise null is allowed. This parameter must be set if single XML documents are either retrieved from Tamino or are created as TXMLObject instances.</I><BR><BR>In my case a null is suffient. see below <BR><BR><BR>3) <B> saxElementClass </B> -<I> a class implementing the TSAXElement interface. This parameter is required if the elementDefaultHandler parameter is not null, otherwise null is allowed. This parameter must be set if XML documents resulting from a Tamino query are to be processed.</I><BR><BR>Thats exactly what I need.  So I need this parameter to be set.  But how...<BR><BR>4)<B> documentDefaultHandler </B> - <I> an instance of a class extending the TSAXDocumentDefaultHandler abstract class. This parameter is optional, but required if single XML documents are either retrieved from Tamino or are created as TXMLObject instances. If this parameter is specified, the saxDocumentClass parameter must also be set.</I><BR><BR>Well since it is optional and in my case not needed I set it to null.<BR><BR>5) <B>elementDefaultHandler</B> - <I> an instance of a class extending the TSAXElementDefaultHandler abstract class. This parameter is optional, but required if XML documents resulting from a Tamino query are to be processed. If this parameter is specified, the saxElementClass parameter must also be set.</I><BR><BR>Ok I need this one too.  But again I don't know how to do that.  <BR><BR><pre class="ip-ubbcode-code-pre">TSAXElementDefaultHandler elementDefaultHandler;</pre><BR><BR>This would be a start but I really don't know how to initialize that...<BR><BR><BR>Regarding your example, I don't understand this part:<BR><pre class="ip-ubbcode-code-pre">
// Instantiate the default handler that processes the sax events
YOUREXAMPLEDefaultHandler = new YOUREXAMPLEDefaultHandler();

// Instantiate the document and element event handlers each of which 
// delegates its events to the greetingDefaultHandler
elDefHandler = new ElementDefaultHandler( YOUREXAMPLEDefaultHandler );

Would you like to elaborate that more to me please?

Thank you very much,

Houman Khorasani
University of Wisconsin Platteville

Actually I found this example here.

http://tamino.forums.softwareag.com/viewtopic.php?p=2322

It’s a pitty I didn’t find it earlier. I just checked it on the fly and I understand the idea behind it. I will study it deeper tomorrow and will let you guys know if it worked out or not. Right now it’s again 4 in the morning.

Have a good night

Houman Khorasani
University of Wisconsin Platteville

kool


With Regards,
Sonal.

Hello evertone,

I have still problems. I have downloaded the example and tried to implement it in my application.

TSAXObjectModel saxObjectModel = new TSAXObjectModel( "MessageSAXObjectModel" , Message.class , Message.class ,	docDefHandler , elDefHandler );

// Do the object model registration.
TXMLObjectModel.register( saxObjectModel );
// Obtain the concrete TXMLObjectAccessor with an underyling JDOM object model
accessor = connection.newXMLObjectAccessor( TAccessLocation.newInstance( collection ) , saxObjectModel );

TXQuery xquery = TXQuery.newInstance(query);

TResponse response = accessor.xquery(xquery,100);

TXMLObjectIterator iterator = response.getXMLObjectIterator();
</pre><BR><BR>The iterator remains empty.  That means it didn't parse right.  I just took the example as it was.  I went through it, there is nothing specific in the classes which needs adjustment, why doesn't it parse right? <BR><BR>I guess the problem is in the Message.class within the line <BR><pre class="ip-ubbcode-code-pre">
TSAXObjectModel saxObjectModel = new TSAXObjectModel( "MessageSAXObjectModel" , Message.class , Message.class ,	docDefHandler , elDefHandler );

I don’t think thet the ElementDefaulthandler needs any adjustment. If there is somethig it should be the Message.class.

I appreciate any comment, I really need to make the API work efficiently with any kind of XQuery.

Thank you very much

Houman Khorasani
University of Wisconsin Platteville

[This message was edited by Houman Khorasani on 18 May 2004 at 22:56.]

Hi Nilesh,

I have also tried it with DOM. Your line of work compiles and seems to be alright if I use a XQuery with a smaller result.

for $i in input()/data
where ($i/row/dat = '20000801000000')
return count($i/row)
</pre><BR><BR>It returns just one count result and it works fine.<BR><BR>But if I use this XQuery:<BR> <BR><pre class="ip-ubbcode-code-pre">
for $i in input()/data/row
where ($i/dat = '20000801000000')
return 		
<ROW>
{ $i/tick }
</ROW>

I get a Tamino access failure (INOXYE9291, Transaction aborted because it has taken too long)

I have already increased the Maximum duration time of the Server AND maximum duration time of XML both from 300 to 900 seconds.

The result is about 80000 … tags. The interestig part is that I have already set a limit to 1000 recordssets:

TResponse response = accessor.xquery(xquery,1000);

In this case it shouldn’t matter if I am using DOM or SAX, since 1000 should be small enough to fit into the memory easily. It should also work with DOM but it doesn’t. So whats wrong here?

Do you have any idea, why it takes so long to perform a query?

Thank you

Houman Khorasani
University of Wisconsin Platteville

hi houman,

This topic had been covered in the community before, so i’ll make do with the short answer: the ‘limit’ of 1000 you are stating does NOT limit the amount of documents that you will receive as the result set, but will let the underlying TaminoAPI4J fetch the resulting documents in blocks of 1000 documents (assisted by a cursor in the xml server). however, this is totally transparent, so as soon as the 1000th document is read, a subsequent next() operation will fetch the next cursor segment and you might not even notice the difference.

i hope this helps,

andreas f.

Houman,

Are you using Transaction in your code? Because the method xquery(TXQuery xquery, int quantity) should be used in a local transaction mode. This is due to the fact that it makes use of a Tamino cursor which can only be used in a transactional context.

Please check.

Thanks

Nilesh Y.

Nilesh,

It is already in local transaction mode. The error is still there.

Regards
Houman Khorasani
University of Wisconsin Platteville

Hi Houman,

Can you please send your code where you are handling the connection and transactions so that I can have a look on it?

Thanks

regards

Nilesh Y.

Hi Nilesh,

Sure.

private TConnection connection = null;

TConnectionFactory connectionFactory = TConnectionFactory.getInstance();

connection = connectionFactory.newConnection(databaseURI);
//accessor = connection.newXMLObjectAccessor(TAccessLocation.newInstance(collection),TDOMObjectModel.getInstance());

accessor = connection.newXMLObjectAccessor(TAccessLocation.newInstance(collection), TJDOMObjectModel.getInstance());

TXQuery xquery = TXQuery.newInstance(query);
TLocalTransaction tloc = connection.useLocalTransactionMode();
TResponse response = accessor.xquery(xquery, 5000);  <--It already crashes here!
TXMLObjectIterator iterator = response.getXMLObjectIterator();

Regards

Houman Khorasani
University of Wisconsin Platteville

Houman,

Try to set the following after getting the connection object.

connection.setIsolationDegree(TIsolationDegree.UNCOMMITTED_DOCUMENT);

It sets the lock mode associated to this accessor object to the value “unprotected” in order to avoid any locks at all.
It is in fact very close to the minimum possible degree of locking at all. Processing with very high performance can therefore be expected.

Also kindly go through the documentation regarding cursors and transactions of Tamino.

Please let me know the result.

Thanks

regards

Nilesh Y.

Hi Nilesh,

I tried it. It won’t work.
Don’t know what to say. But I appreciate your help.

Regards

Houman Khorasani
University of Wisconsin Platteville