Our group has run into a limitation with xquery that has pushed us to create a preprocessor for xquery and the one downfall to this approach is that our tools (eclipse with oxygen) now don't support the xqueries as-is. We were wondering if there is a recommended way for creating a plugin (java) for oxygen that can preprocess the xquery prior to passing it off to the debugger or xquery instance (such as eXist). Thank you, Paul Ryan Email pryan@infotrustgroup.com<mailto:pryan@infotrustgroup.com>
Paul Ryan wrote: Hi,
Our group has run into a limitation with xquery that has pushed us to create a preprocessor for xquery
I don't know if what you ask for is possible or not (I am sure the oXygen team will respond you on that matter,) but I wonder what kind of limitations you ran into, and how you use a preprocessor to solve them. Would you mind to say a few words about this, just out of interest? Regards, -- Florent Georges http://www.fgeorges.org/
Hello, Do you mean your XQuery queries are reported as invalid in Oxygen because they are not standard W3C XQuery? You can set the validation engine to be the same as the processing engine such as eXist from Options -> Preferences -> XML -> XSLT-FO-XQuery -> XQuery or by creating a validation scenario. Do you mean you want to send to the debugging engine a modified version of the XQuery file that is edited in a file tab of Oxygen? That is not possible now. Can you give one or two examples of the limitations and the needed preprocessing? Regards, Sorin Florent Georges wrote:
Our group has run into a limitation with xquery that has pushed us to create a preprocessor for xquery
I don't know if what you ask for is possible or not (I am sure the oXygen team will respond you on that matter,) but I wonder what kind of limitations you ran into, and how you use a preprocessor to solve them.
Sure the limitation has to do with working with Java and XQuery together. When paired, to pass data down to the XQuery, it ends up being passed as strings which with an engine like eXist the data needs to be evaluated with a special utility extension. This presents a couple of problems first you have to construct your queries using a string which can cause issues, second you are forced into using a database specific extension such as the http://exist-db.org/xquery/util module. We want to avoid this so we've come up with a preprocessor that does a token replace prior to passing off to the XQuery engine. Our tokens are in the form "\$\{([a-zA-Z0-9]+(\:[a-zA-Z0-9]+)?)\}". I'm not against rewriting this custom pre-processor as a plug-in to oXygen however I didn't immediately see a way to do that. Sorin, What do you mean by creating a validation scenario? Are you referring to the scenarios for the db connection + query and if so how would I attach a custom pre-processor to this; or do you mean I should attach a "custom engine" and if so can you point me to a reference on how to build one of these. Thanks again for your help, -- Paul Ryan -----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Monday, January 12, 2009 1:43 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery Hello, Do you mean your XQuery queries are reported as invalid in Oxygen because they are not standard W3C XQuery? You can set the validation engine to be the same as the processing engine such as eXist from Options -> Preferences -> XML -> XSLT-FO-XQuery -> XQuery or by creating a validation scenario. Do you mean you want to send to the debugging engine a modified version of the XQuery file that is edited in a file tab of Oxygen? That is not possible now. Can you give one or two examples of the limitations and the needed preprocessing? Regards, Sorin Florent Georges wrote:
Our group has run into a limitation with xquery that has pushed us to create a preprocessor for xquery
I don't know if what you ask for is possible or not (I am sure the oXygen team will respond you on that matter,) but I wonder what kind of limitations you ran into, and how you use a preprocessor to solve them.
_______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
Paul Ryan wrote:
Sure the limitation has to do with working with Java and XQuery together. When paired, to pass data down to the XQuery, it ends up being passed as strings which with an engine like eXist the data needs to be evaluated with a special utility extension.
I am sorry to use oXygen band-width for what seems to be a general XQuery question, but I am really intrigued: do you mean the usual way of passing variable to a query (using external variables) does not work with eXist, from Java? Because if it is just a matter of passing strings to the query, I would personally rather use external variables in the query, set them from the calling Java code, and not use any pre-processing phase; in other words always use only standard queries. But I guess I missed something. Regards, -- Florent Georges http://www.fgeorges.org/
What you suggest poses a unique problem however because the java would have to know ahead of time (prior to calling xquery) if the string being passed will be used in an evaluation. This limitation with external variables of having to pre-evaluate passed elements is a large hole for us. The middle tier that is passing the query arguments doesn't know what variables will be used for evaluation however the XQuery string/document has this information. To ensure that string variables that need to be evaluated can be evaluated we opted to create a token preprocessor vs. building logic into the java to predetermine if it should evaluate the variable to an element. As an example parsing the following has this issue: Exist xquery - xquery version "1.0"; import module namespace util="http://exist-db.org/xquery/util"; declare variable $path as xs:string external; let $query := concat('for $x in collection()',$path, ' return $x') return util:eval($query) Desired xquery - xquery version "1.0"; declare variable $path as xs:string external; for $x in collection()$path return $x Token syntax for achieving desired xquery- xquery version "1.0"; for $x in collection()${path} return $x Result of preprocessing on tokens- xquery version "1.0"; for $x in collection()/mypath[@id='myid'] return $x Our preprocessor handles a little more than is shown but the example above is the general idea behind our preprocessor. -- Paul Ryan -----Original Message----- From: Florent Georges [mailto:lists@fgeorges.org] Sent: Monday, January 12, 2009 10:55 AM To: Paul Ryan; oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery Paul Ryan wrote:
Sure the limitation has to do with working with Java and XQuery together. When paired, to pass data down to the XQuery, it ends up being passed as strings which with an engine like eXist the data needs to be evaluated with a special utility extension.
I am sorry to use oXygen band-width for what seems to be a general XQuery question, but I am really intrigued: do you mean the usual way of passing variable to a query (using external variables) does not work with eXist, from Java? Because if it is just a matter of passing strings to the query, I would personally rather use external variables in the query, set them from the calling Java code, and not use any pre-processing phase; in other words always use only standard queries. But I guess I missed something. Regards, -- Florent Georges http://www.fgeorges.org/
Hello, If you need to compose an XPath expression with the runtime value of a parameter of the query then you need what is called a dynamic evaluation of an XPath expression. The standard XQuery language does not provide such an evaluation which seems to be due (some experts say that) to the difficulty of determining the evaluation context. That means you should use an extension function implemented in the specific XQuery engine that runs your queries, for example util:eval() in eXist or saxon:evaluate() in Saxon 9 instead of implementing your own mechanism of dynamic evaluation (your token pre-processor) and this is why Oxygen does not provide such a mechanism. The eXist extension function is explained at: http://demo.exist-db.org/exist/xquery.xml?q=.%2F%2Fsection%2Ftitle[.%20%26%3D%20%27function%20library%27]%20or%20.%2F%2Fpara[.%20%26%3D%20%27function%20library%27]&id=1.3.3#N104A0 Regards, Sorin Paul Ryan wrote:
What you suggest poses a unique problem however because the java would have to know ahead of time (prior to calling xquery) if the string being passed will be used in an evaluation. This limitation with external variables of having to pre-evaluate passed elements is a large hole for us. The middle tier that is passing the query arguments doesn't know what variables will be used for evaluation however the XQuery string/document has this information. To ensure that string variables that need to be evaluated can be evaluated we opted to create a token preprocessor vs. building logic into the java to predetermine if it should evaluate the variable to an element.
As an example parsing the following has this issue:
Exist xquery - xquery version "1.0";
import module namespace util="http://exist-db.org/xquery/util"; declare variable $path as xs:string external;
let $query := concat('for $x in collection()',$path, ' return $x') return util:eval($query)
Desired xquery - xquery version "1.0";
declare variable $path as xs:string external;
for $x in collection()$path return $x
Token syntax for achieving desired xquery- xquery version "1.0";
for $x in collection()${path} return $x
Result of preprocessing on tokens- xquery version "1.0";
for $x in collection()/mypath[@id='myid'] return $x
Our preprocessor handles a little more than is shown but the example above is the general idea behind our preprocessor.
-- Paul Ryan
Sorrin, I understand the delima you present and we've put sufficient limitations on preprocessor to ensure proper context. In your first email you stated that I would override the validation scenario such as is done by exist. Is there a special API I should follow or just the standard xmldb-api from 2002 should do? Thanks, -- Paul Ryan -----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Wednesday, January 14, 2009 8:24 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery Hello, If you need to compose an XPath expression with the runtime value of a parameter of the query then you need what is called a dynamic evaluation of an XPath expression. The standard XQuery language does not provide such an evaluation which seems to be due (some experts say that) to the difficulty of determining the evaluation context. That means you should use an extension function implemented in the specific XQuery engine that runs your queries, for example util:eval() in eXist or saxon:evaluate() in Saxon 9 instead of implementing your own mechanism of dynamic evaluation (your token pre-processor) and this is why Oxygen does not provide such a mechanism. The eXist extension function is explained at: http://demo.exist-db.org/exist/xquery.xml?q=.%2F%2Fsection%2Ftitle[.%20%26%3D%20%27function%20library%27]%20or%20.%2F%2Fpara[.%20%26%3D%20%27function%20library%27]&id=1.3.3#N104A0 Regards, Sorin Paul Ryan wrote:
What you suggest poses a unique problem however because the java would have to know ahead of time (prior to calling xquery) if the string being passed will be used in an evaluation. This limitation with external variables of having to pre-evaluate passed elements is a large hole for us. The middle tier that is passing the query arguments doesn't know what variables will be used for evaluation however the XQuery string/document has this information. To ensure that string variables that need to be evaluated can be evaluated we opted to create a token preprocessor vs. building logic into the java to predetermine if it should evaluate the variable to an element.
As an example parsing the following has this issue:
Exist xquery - xquery version "1.0";
import module namespace util="http://exist-db.org/xquery/util"; declare variable $path as xs:string external;
let $query := concat('for $x in collection()',$path, ' return $x') return util:eval($query)
Desired xquery - xquery version "1.0";
declare variable $path as xs:string external;
for $x in collection()$path return $x
Token syntax for achieving desired xquery- xquery version "1.0";
for $x in collection()${path} return $x
Result of preprocessing on tokens- xquery version "1.0";
for $x in collection()/mypath[@id='myid'] return $x
Our preprocessor handles a little more than is shown but the example above is the general idea behind our preprocessor.
-- Paul Ryan
_______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
Hello, No, the validation scenario does not help you to insert your pre-processing step before sending the XQuery file to the eXist database. The database always receives the file content that is displayed in the Oxygen editor panel. You can use a scenario only for setting eXist as the validating engine instead of the default one (Saxon 9) so that the validation action does not report the eXist extension as an error and I suggested that before understanding what you mean by queries not supported "as is". The validation scenario is explained in the User Manual: http://www.oxygenxml.com/doc/ug-oxygen/validation-scenario.html Regards, Sorin Paul Ryan wrote:
Sorrin,
I understand the delima you present and we've put sufficient limitations on preprocessor to ensure proper context. In your first email you stated that I would override the validation scenario such as is done by exist. Is there a special API I should follow or just the standard xmldb-api from 2002 should do?
Thanks,
-- Paul Ryan
-----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Wednesday, January 14, 2009 8:24 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery
Hello,
If you need to compose an XPath expression with the runtime value of a parameter of the query then you need what is called a dynamic evaluation of an XPath expression. The standard XQuery language does not provide such an evaluation which seems to be due (some experts say that) to the difficulty of determining the evaluation context. That means you should use an extension function implemented in the specific XQuery engine that runs your queries, for example util:eval() in eXist or saxon:evaluate() in Saxon 9 instead of implementing your own mechanism of dynamic evaluation (your token pre-processor) and this is why Oxygen does not provide such a mechanism.
The eXist extension function is explained at:
Regards, Sorin
Paul Ryan wrote:
What you suggest poses a unique problem however because the java would have to know ahead of time (prior to calling xquery) if the string being passed will be used in an evaluation. This limitation with external variables of having to pre-evaluate passed elements is a large hole for us. The middle tier that is passing the query arguments doesn't know what variables will be used for evaluation however the XQuery string/document has this information. To ensure that string variables that need to be evaluated can be evaluated we opted to create a token preprocessor vs. building logic into the java to predetermine if it should evaluate the variable to an element.
As an example parsing the following has this issue:
Exist xquery - xquery version "1.0";
import module namespace util="http://exist-db.org/xquery/util"; declare variable $path as xs:string external;
let $query := concat('for $x in collection()',$path, ' return $x') return util:eval($query)
Desired xquery - xquery version "1.0";
declare variable $path as xs:string external;
for $x in collection()$path return $x
Token syntax for achieving desired xquery- xquery version "1.0";
for $x in collection()${path} return $x
Result of preprocessing on tokens- xquery version "1.0";
for $x in collection()/mypath[@id='myid'] return $x
Our preprocessor handles a little more than is shown but the example above is the general idea behind our preprocessor.
-- Paul Ryan
_______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user _______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
Ok, that being the case it sound like my only option is to make my own extension of eXist. I'll go down that path (unless I'm misunderstanding). Thanks, -- Paul Ryan -----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Monday, January 19, 2009 2:24 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery Hello, No, the validation scenario does not help you to insert your pre-processing step before sending the XQuery file to the eXist database. The database always receives the file content that is displayed in the Oxygen editor panel. You can use a scenario only for setting eXist as the validating engine instead of the default one (Saxon 9) so that the validation action does not report the eXist extension as an error and I suggested that before understanding what you mean by queries not supported "as is". The validation scenario is explained in the User Manual: http://www.oxygenxml.com/doc/ug-oxygen/validation-scenario.html Regards, Sorin Paul Ryan wrote:
Sorrin,
I understand the delima you present and we've put sufficient limitations on preprocessor to ensure proper context. In your first email you stated that I would override the validation scenario such as is done by exist. Is there a special API I should follow or just the standard xmldb-api from 2002 should do?
Thanks,
-- Paul Ryan
-----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Wednesday, January 14, 2009 8:24 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery
Hello,
If you need to compose an XPath expression with the runtime value of a parameter of the query then you need what is called a dynamic evaluation of an XPath expression. The standard XQuery language does not provide such an evaluation which seems to be due (some experts say that) to the difficulty of determining the evaluation context. That means you should use an extension function implemented in the specific XQuery engine that runs your queries, for example util:eval() in eXist or saxon:evaluate() in Saxon 9 instead of implementing your own mechanism of dynamic evaluation (your token pre-processor) and this is why Oxygen does not provide such a mechanism.
The eXist extension function is explained at:
Regards, Sorin
Paul Ryan wrote:
What you suggest poses a unique problem however because the java would have to know ahead of time (prior to calling xquery) if the string being passed will be used in an evaluation. This limitation with external variables of having to pre-evaluate passed elements is a large hole for us. The middle tier that is passing the query arguments doesn't know what variables will be used for evaluation however the XQuery string/document has this information. To ensure that string variables that need to be evaluated can be evaluated we opted to create a token preprocessor vs. building logic into the java to predetermine if it should evaluate the variable to an element.
As an example parsing the following has this issue:
Exist xquery - xquery version "1.0";
import module namespace util="http://exist-db.org/xquery/util"; declare variable $path as xs:string external;
let $query := concat('for $x in collection()',$path, ' return $x') return util:eval($query)
Desired xquery - xquery version "1.0";
declare variable $path as xs:string external;
for $x in collection()$path return $x
Token syntax for achieving desired xquery- xquery version "1.0";
for $x in collection()${path} return $x
Result of preprocessing on tokens- xquery version "1.0";
for $x in collection()/mypath[@id='myid'] return $x
Our preprocessor handles a little more than is shown but the example above is the general idea behind our preprocessor.
-- Paul Ryan
_______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user _______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
From your example it seemed that the util:eval() built-in extension of eXist was enough for the XPath evaluation that you needed. Other option is to split the query in two steps: the first step is an XQuery that evaluates the part of the XPath expression that is passed as parameter in your example and the second step is an XQuery applied to the result of the first step to do the main task of your query. Regards, Sorin Paul Ryan wrote:
Ok, that being the case it sound like my only option is to make my own extension of eXist. I'll go down that path (unless I'm misunderstanding).
Thanks,
-- Paul Ryan
-----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Monday, January 19, 2009 2:24 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery
Hello,
No, the validation scenario does not help you to insert your pre-processing step before sending the XQuery file to the eXist database. The database always receives the file content that is displayed in the Oxygen editor panel. You can use a scenario only for setting eXist as the validating engine instead of the default one (Saxon 9) so that the validation action does not report the eXist extension as an error and I suggested that before understanding what you mean by queries not supported "as is".
The validation scenario is explained in the User Manual:
http://www.oxygenxml.com/doc/ug-oxygen/validation-scenario.html
Regards, Sorin
Paul Ryan wrote:
Sorrin,
I understand the delima you present and we've put sufficient limitations on preprocessor to ensure proper context. In your first email you stated that I would override the validation scenario such as is done by exist. Is there a special API I should follow or just the standard xmldb-api from 2002 should do?
Thanks,
-- Paul Ryan
-----Original Message----- From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Sorin Ristache Sent: Wednesday, January 14, 2009 8:24 AM To: oxygen-user@oxygenxml.com Subject: Re: [oXygen-user] Preprocessing Xquery
Hello,
If you need to compose an XPath expression with the runtime value of a parameter of the query then you need what is called a dynamic evaluation of an XPath expression. The standard XQuery language does not provide such an evaluation which seems to be due (some experts say that) to the difficulty of determining the evaluation context. That means you should use an extension function implemented in the specific XQuery engine that runs your queries, for example util:eval() in eXist or saxon:evaluate() in Saxon 9 instead of implementing your own mechanism of dynamic evaluation (your token pre-processor) and this is why Oxygen does not provide such a mechanism.
The eXist extension function is explained at:
Regards, Sorin
Paul Ryan wrote:
What you suggest poses a unique problem however because the java would have to know ahead of time (prior to calling xquery) if the string being passed will be used in an evaluation. This limitation with external variables of having to pre-evaluate passed elements is a large hole for us. The middle tier that is passing the query arguments doesn't know what variables will be used for evaluation however the XQuery string/document has this information. To ensure that string variables that need to be evaluated can be evaluated we opted to create a token preprocessor vs. building logic into the java to predetermine if it should evaluate the variable to an element.
As an example parsing the following has this issue:
Exist xquery - xquery version "1.0";
import module namespace util="http://exist-db.org/xquery/util"; declare variable $path as xs:string external;
let $query := concat('for $x in collection()',$path, ' return $x') return util:eval($query)
Desired xquery - xquery version "1.0";
declare variable $path as xs:string external;
for $x in collection()$path return $x
Token syntax for achieving desired xquery- xquery version "1.0";
for $x in collection()${path} return $x
Result of preprocessing on tokens- xquery version "1.0";
for $x in collection()/mypath[@id='myid'] return $x
Our preprocessor handles a little more than is shown but the example above is the general idea behind our preprocessor.
-- Paul Ryan
oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user _______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user _______________________________________________ oXygen-user mailing list oXygen-user@oxygenxml.com http://www.oxygenxml.com/mailman/listinfo/oxygen-user
I am sorry, I mean split the query in two steps: the first step is an XQuery that generates other XQuery in which the $path parameter is expanded (replaced with the value of the parameter) and the second step runs the generated XQuery. Regards, Sorin Sorin Ristache wrote:
From your example it seemed that the util:eval() built-in extension of eXist was enough for the XPath evaluation that you needed. Other option is to split the query in two steps: the first step is an XQuery that evaluates the part of the XPath expression that is passed as parameter in your example and the second step is an XQuery applied to the result of the first step to do the main task of your query.
Regards, Sorin
Paul Ryan wrote:
Ok, that being the case it sound like my only option is to make my own extension of eXist. I'll go down that path (unless I'm misunderstanding).
Thanks,
-- Paul Ryan
participants (3)
-
Florent Georges -
Paul Ryan -
Sorin Ristache