X-Application Version: 4.1.1.4
Tamino Version : 4.1.1
Platform : WinXP
WebContainer : Tomcat 4.1.18
JDK Version : 1.3.1_06


Hello,

I use a large database with more than 325.000 documents.

When I query data with x-query via the Interactive Interface I get the results within seconds.

When I do the same query via my X-Application it takes ca. 15 minutes until I get an error message, which says that the XML maximum query duration exceeded.

On a databse with only 30.000 documents the same query with the same X-Application finishes successfully within 5 Minutes.

I attach my Tamino-Schema and an example for a structure.xml to this posting.

Is the bad performance caused by the largeness of my schema and the database? Are there possibilities to increase the performance of X-Application?

Best regards
Thorsten
schema_structure.zip (4.87 KB)

Hello Thorsten,

are you using X-Query or XQuery?

What the differenec?

XQuery is used within action tags or directcommand tags with the value ‘xquery’ for the ‘type’ attribute. The query expression looks like

for $x input()/… return $x


X-Query is used via type=“query” within the tags metioned above.

How to increase the performance?

For XQuery: The support for querying over a large set of documents does not have a good performance. To improve the performance is a task we had to solve for a next version.
As a performance workaround you could use a X-Query expression if its possible to transform your query.

For X-Query: We should check the communication between Tamino Server and X-Application for your application. This protocol would help us to find the performance lack of X-Application and we could look for a workaround.

Many thanks.

Bye,
Christian.

Hello Christian,

I produced the X-Application with the Generator and it automatically set the type attribute of the form fields to “xquery”.

How can I change xquery to x-query?

Do I have to change the java server pages manually or is it possible to change the xapp settings to favour x-query instead of xquery (and finally regenerate my xapplication)?

Best regards
Thorsten

Hello,

no there is no property or switch you can set the underlying query language.

My propose is to change the query language from XQuery to X-Query only for queries that have a bad performance.

How to change the query languange?

Normally, you have a search page where the query is sent via xapp:action.

(1) You have to set the attribute ‘type’ from ‘xquery’ to ‘query’.

(2) The attribute ‘arg’ refers to a variable that contains the query expression. For the generated page this is an XQuery expression of the form

for $x in input()/doctype${FILTER_…$} return $x.

An equivalent X-Query expression would be

/doctype${FILTER_…$}

The doctype is the root element of your documents (e.g. ‘Property’). So, the only thing to do is to cut of

for $x in input()

at the beginning of the XQuery and

return $x

at its end.

If this does not work, please send me your application inclusive schema and I will look for a solution.

Bye,
Christian.

Ok I have this xquery that list all the titels of my xml arcive by listing 5 at the time in a table. I have like 300 xml documents on 1mb eache and I have a really bad performance.
If I do a recuest like this directly to the database I get a fast replay

 http://192.168.3.11/tamino/dtbook/dtbook?_xql=/dtbook/head/title </pre><BR><BR>What is wrong ???? or how can I get better speed<BR><BR>but if I use this recuest in xapplication it takes for ever<BR><pre class="ip-ubbcode-code-pre"> <xapp:form>
<xapp:if condition="notexist" document="browseWithNav" select="/dtbook">
<xapp:directcommand collection="dtbook" type="xquery" document="browseWithNav" arg="for $x in input()/dtbook where tf:getInoId($x)>=1 return $x" />
</xapp:if>
</xapp:form>  

X-Application Version: 4.1.1,
Tamino Version : 4.1.1,
Platform : Win2k
WebContainer : Tamocat 4.1.1,
JDK Version : 1.4.1,

[This message was edited by DBB on 30 Apr 2003 at 11:10.]

Hello,
you compare a native X-Query (Tamino 3.1 Query language) with an XQuery (Tamino 4.1 Query language) plus x-application.
To find out the main originator of the probelm, could you try a native XQuery (that means without x-application)? You could use the Interactive Interface.

Regards, Harald

[This message was edited by Harald Wolf on 01 May 2003 at 22:14.]

Ok when I do this in interactive interface i get all the documents and it looks fast but the only problem then I can see is that x-application loads all documents in the memory before it can have a look at the nodes. So therefor it gets like 300mb in memory so it becomes slow. or am I wrong here.
is there a faster way to get all ID from my documents and still be able to use xapp:brows???

Hello Christian,

I have changed the query language from XQuery to X-Query and now it works (even on the database with over 320.000 documents) and is much faster than before.

Thank you very much for your help.

Best regards
Thorsten

Hello,

ok. Good to hear that the workaround helps you. Therefore, alos for the performance problem DBB the switch to X-Query could help to get a faster access to the documents.

Could you (DBB) change your directcommand in the following way?

  <xapp:directcommand collection="dtbook" type="query"  document="browseWithNav" arg="/dtbook" />
</pre><BR><BR>This query returns all 'dtbook' documents, but only the number of documents specified by the pagesize (default = 10) are request when the browse tag is evaluated.<BR><BR>Another possibility is to use XQuery. Since your are interested in the titles of your documents your can reduce the transferred data. The XQuery for getting the titles:<BR><BR><pre class="ip-ubbcode-code-pre">
<xapp:directcommand collection="dtbook" type="xquery"  document="browseWithNav" arg="for $x in input()/dtbook return $x/head/title" />
</pre><BR><BR>To display the titles use<BR><BR><pre class="ip-ubbcode-code-pre">
<xapp:browse ...>
<xapp:display select='.' />
</xapp:browse>

I hope, that one of these approaches helps to solve your performance problem.

Bye,
Christian.

Ok I will be more clear this time I hope Christian I have done all your good advice but it still seams slow here so I will now post the hole jsp page and you can see if it is something wrong in it.
But remember that I am loading 300 1 mb documents when i do request and that is alot in one go I need the ino:id of the document and the title and i would want to brows them whit a display of 5 each Also all my documents are deep like 7 elements deep on some nodes.
send.jsp (3.31 KB)

The error message I get from the x-application in the error page is

Exception Type: ApplicationException    
Stacktrace:


XException: id = 999
java.lang.OutOfMemoryError

hope this means some

Hello,

I changed the directcommand to type=“query” and I set the internal variable :pagesize which controls the number of documents initially load by the cursor.

In this way, only 2 documents should be transferred per request.

If it is too slow with my modification, please send me a zip files with your schema and a test document. Then, we will check if we are able to improve the performance bye modifying the source code of X-Application.

Bye,
Christian.
send-cft.jsp (3.3 KB)

Hello,

Did you use teh JSP I sent?

If yes, set the pagesize attribute of the browse tag.

<xapp:browse pagesize="2" ...>
...
</xapp:browse>

Bye,
Christian.

Yes the pagesize variabel was probely what did it here. Becusse i was using the variable in the brows and not in the request so therefor i got like 300 documents in a request but only displayed 5. So silly me and thanks a lot for this nice replay and fix

PS: I can tell you that the next is twice as fast as previous That might be becusse of the way the xml parsing and that indexing is done

[This message was edited by DBB on 06 May 2003 at 12:26.]

Hello Christian,

unfortunately I have a new problem since I have changed the query language from xquery to x-query in the way you told me.

I recieve errors when I try to commit documents after modifying or creating them.

The errormessage after commiting the modified document is as follows: “Update Conflict: Another user has modfied the document “for $x in input()/dblp where tf:getInoId($x)=4 return $x”.”

I attach the errors I recieve to this posting.

Thanks for your help in advance.

Best regards
Thorsten
error.zip (3.52 KB)

Hello,

I checked the stacktrace, but couldn’t get enaough information from it to find out the reason for this error. What could see is:

- a new document has to be inserted.
- the insert action itself is successful.
- but reading the metadata (timestamp, doctype, …) that X-Application requires to maintain the documents internally, fails.

The error message is

Tamino access failure (INOXME8504, XML maximum query duration exceeded)

I will have a talk with my colleagues. Perhaps, they have an idea how to solve this problem.

To get more information about your situation, I would please you to protocoll the communication between X-Application and Tamino. I posted a description how to install a sniffer to trace the HTTP communication. You will find the post following the link

Protocolling the HTTP transfer between Tamino and X-Application

Many thanks.

Bey,
Christian.

Hello Christian,

I have sniffered the http-communication for

1.) the commitment of a modified document and
2.) the commitment of creating a new document.

I attach the protocol files to this posting.

By the way, it takes ca. 5 minutes to get the modify.jsp after pressing the modify-button and further 10 minutes to get the error-message after pressing the commit-button.

In addition I attach the error-message of commiting the modified document because it looks a bit different this time.

Finally I just want to remind you that the error is connected to the change of the query language from xquery to x-query.

Thanks for your help.

Best regards
Thorsten
error_protocol.zip (12.2 KB)

Hello,

many thnaks for tracing the HTTP communication. I checked the protocol for the insert scenario: The xquery for reading the new document times out. I forwarded your protocol to the Tamino developers who will check this situation.

This is the current status. I will inform you as sone as a solution is found. Perhaps, you will be contacted by my colleagues if they need further information.

Many thanks for the trace.

Bye,
Christian.

I think the main cause of all these difficuties lies on a deeper level - in Tamino. New XQuery optimization might be better. Seems like all tests have been done with only 10 documents.

I also tried to generate an aplication for doctype with 380 000 instances. The result is almost the same.

FOR SEARCH PAGE:

If no criteria is specified on the page, the query to Tamino will have a simple form:
for $x in input()/sklgaz return $x

If doctype is large, this query can take several minutes, even if :pagesize variable (or cursor size) is set to just 1 doc. Seems like Tamino calculates whole result regardless of cursor size (I’ve already created a support request on this topic, waiting for response).

So you could either prevent end user from returning large result set or use old tried X-Query.

FOR CREATE AND UPDATE PAGES:

I took a look at the logs you posted and ran the following query with TII:
declare namespace tf = “http://namespaces.softwareag.com/tamino/TaminoFunction”
for $x in input()/sklgaz where tf:getInoId($x)=1 return $x
And again I have several minutes for coffee. Since this query is a part of reading timestamp and other metadata, there is nothing surprising that create and update requests timed out.

[This message was edited by Alexander on 18 May 2003 at 23:32.]

Hello,

we have analyzed the problem that NSDB sent us. For large sets of documents, browsing in X-Application for XQuery is inperformant. As a workaround, we proposed to switch to X-Query. To solve the XQuery problem is task for a next version of X-Application.

However, the effect that you can look for a cup of coffee when committing a new / modified document was a problem of the Tamino Server which is fixed for the next released version of Tamino. Since the next release 4.1.4 comes up end of this month (May 2003) I would please you to wait for this new version.

Bye,
Christian.