Content-oriented XML retrieval is an area of Information Retrieval (IR) research that has been receiving increasing interest fuelled by the widespread use of the eXtensible Markup Language (XML), which is becoming a standard document format on the Web, in Digital Libraries, and Publishing. The continuous growth in XML data sources is matched by increasing efforts in the development of XML IR systems. These systems aim to harness the enriched source of syntactic and semantic information that XML markup provides.Current work in XML IR focuses on exploiting the available structural information in documents to implement a more focused retrieval strategy and return document components, the so-called XML elements - instead of complete documents - in response to a user query. This focused retrieval approach is of particular benefit for collections containing long documents or documents covering a wide variety of topics (e.g. books, user manuals, legal documents, etc.), where the users' effort to locate relevant content can be reduced by direct- ing them to the most relevant parts of the documents. For example, in response to a user query on a collection of scientific articles marked-up in XML, an XML IR system may return e.g., a mixture of paragraph, section, article elements, that have been estimated as to best answer the user's query. Implementing this, more focused, retrieval paradigm means that an XML retrieval system needs not only to find relevant information in the XML documents, but also determine the appropriate level of granularity to be returned to the user. In addition, the relevance of a retrieved component is dependent on meeting both content and structural conditions.

Evaluating the effectiveness of XML retrieval systems, hence, requires a test collection where the relevance assessments are provid- ed according to a relevance criterion, which takes into account the imposed structural aspects. In 2002, the Initiative for the Evalua- tion of XML Retrieval (INEX) started to address these issues. INEX has a strong international character; participants from over 80 organisations, distributed across Europe, America, Australia, Asia, and Middle-East have so far contributed to INEX. The aim of the INEX initiative is to establish an infrastructure and provide means, in the form of a large XML test collection and appropriate scoring methods, for the evaluation of content-oriented XML retrieval systems.

Evaluating retrieval effectiveness is typically done by using test collections assembled specifically for evaluating particular retrieval tasks. A test collection usually consists of a set of documents, a set of user requests (the so-called topics, or queries) and relevance assessments of the documents with respect to the queries. The characteristics of traditional test collections have been adjusted in order to appropriately evaluate content-oriented XML retrieval effectiveness: the document collection comprises documents marked up in XML, the topics specify requests related both to the content of the desired XML elements and to their structural properties, and the relevance assessments are made on the XML element level rather than just on the full document level. In addition, relevance is measured in a different way compared to traditional Information Retrieval research, in order to more effectively quantify the systems' ability to return the right granularity of XML elements. A test collection as such has been built as a result of four rounds of the Initiative for the Evaluation of XML Retrieval (INEX 2002, INEX 2003, INEX 2004 and INEX 2005).

In INEX 2006, participating organisations will be able to compare the retrieval effectiveness of their XML retrieval systems and will contribute to the construction of a new XML test collection based on Wikipedia. The test collection will also provide participants a means for future comparative and quantitative experiments. Please note that only participating organisations will have access to the constructed test collection.

INEX test collection

The test collection consists of a set of XML documents, topics and relevance assessments. The topics and the relevance judgments are obtained through a collaborative effort from the participants. Detailed guidelines on the on-line topic submission, retrieval result submission, relevance assessment task, and evaluation metrics will be provided by INEX.

Documents

INEX 2006 uses a document collection made from English documents from Wikipedia. The collection is so far made up of the full-texts, marked-up in XML, of 659,388 articles of the Wikipedia project, covering a hierarchy of 113,483 categories, and totaling more than 60 Gigabytes (4.6 Gigabytes without images). The collection has a structure containing text, more than 300,000 images and some structured part corresponding to the Wikipedia templates (about 5000 different tags). On average an article contains 161.35 XML nodes, where the average depth of an element is 6.72.

Topics

Each participating group will be asked to create a set of candidate topics, which are representative of a range of real user needs over the XML collection. The queries may be content-only (CO) or content-and-structure (CAS) queries, and broad or narrow topic queries. CO queries are free text queries, like those used in TREC, for which the retrieval system should retrieve relevant XML elements of varying granularity, while CAS queries contain explicit structural constraints, such as containment conditions. From the pooled set of candidate topics INEX will select a final set of topics to form part of the INEX test collection

Tasks

The main retrieval task to be performed in INEX is the ad-hoc retrieval of XML documents. In information retrieval literature, ad-hoc retrieval is described as a simulation of how a library might be used, and it involves the searching of a static set of documents using a new set of topics. While the principle is the same, the difference for INEX is that the library consists of XML documents, the queries may contain both content and structural conditions and, in response to a query, arbitrary XML elements may be retrieved from the library. Within the main ad-hoc retrieval task in INEX 2005, three sub-tasks were identified depending on how structural constraints are expressed in queries.

1. In the Content-Only (CO) sub-task, queries ignore the document structure and contain only content-related conditions.
2. An extension of the CO sub-task that includes structural hints is the +S sub-task, where a user may decide to add structural hints to his query to narrow down the number of returned elements resulting from a CO query.
3. In the Content and Structure (CAS) sub-task, structural constraints are explicitly stated in the query and they can refer both to where to look for the relevant elements (i.e. support elements), and what type of elements to return (i.e. target elements). A structural constraint can also be interpreted as strict (i.e. the structural requirements must be followed strictly) or vague (i.e. the structural constraints are interpreted as hints and the main goal is to satisfy the overall information need). Strict and vague interpretations can be applied to both support and target elements, giving a total of four strategies for the CAS subtask.

With regards to evaluation methodology for the ad-hoc track, depending on how we assume that a user would want the output of an XML retrieval system to be, three different strategies were defined and used in INEX 2005. In a focussed strategy, we assume that a user prefers a single element that most exhaustively discusses the topic of the query (most exhaustive element), while at the same time it is most specific only to that topic (most specific element). In a thorough strategy, we assume that a user prefers all highly exhaustive and specific elements, and in a fetch and browse strategy we assume that a user is interested in highly exhaustive and specific elements that are contained only within highly relevant articles.

It is expected that the INEX 2006 ad hoc retrieval task will be based on a combination/selection of the above sub-tasks and strategies, and newly defined ones.

In addition to the main general ad-hoc retrieval task, INEX 2006 will have the following two specific tasks, which continue from INEX 2005:

1. Relevance feedback task
2. Natural query language task

INEX 2006 will continue with the following four tracks that started in previous years:

1. Heterogeneous collection track
2. Interactive track
3. Document mining track
4. Multimedia track

Two additional tracks are planned for INEX 2006:

1. Use case studies track
2. XML Entity Ranking track

Relevance assessments

Relevance assessments will be provided by the participating groups using INEX's on-line assessment system. Each participating organisation will judge around 2 topics, either the topics that they originally created or if these were removed from the final set of topics, then topics that were similar to their original queries or within their expertise. Please note that assessments take about one person week per topic! Participating groups will gain access to the completed INEX test collection only after they have completed their assessment task.

Evaluation

The evaluation of the retrieval effectiveness of the XML retrieval engines used by the participants will be based on the constructed INEX test collection and uniform scoring techniques. Since its launch in 2002, INEX has been challenged by the issue of how to measure an XML information access system's effectiveness. In 2005, INEX adopted a new set of metrics, the eXtended Cumulated Gain (XCG) metrics to support the evaluation of XML retrieval engines, which will also be used in INEX 2006. These new metrics aim to provide an evaluation framework that allows to consider the dependency that exists among XML document components and, in particular, incorporate mechanisms to reward the retrieval of so-called near-misses and to address issues of overlap.

Workshop and proceedings

Participants will be able to present their approaches and final results at the INEX 2006 workshop to be held in December in Dagstuhl. All descriptions of the approaches and results will be published in the INEX workshop pre-proceedings and the Web. Revised papers will be published in the INEX post-workshop final proceedings. As for INEX 2004 and 2005, we expect the INEX final proceedings to be published in the Springer's Lecture Notes in Computer Science (LNCS) series.

Data Handling Agreement

Upon completion of the relevance assessments, participants new to INEX can have access to the previous years test collections. Up to 2004, the INEX corpus was composed of the full-texts, marked up in XML, of 12,107 articles of the IEEE Computer Society's publications from 12 magazines and 6 transactions, covering the period of 1995-2002, and totalling 494 megabytes in size. On average an article contains 1,532 XML nodes, where the average depth of a node is 6.9. An addition of 4,712 articles totalling 241 megabytes in size from 2002-2004 have been added to the collection in 2005, thus reaching a total of 16,819 articles. To date, INEX has a total of 300 topics, most of them with associated relevance assessments. In order to have access to the data designated as the IEEE Computer Society XML Retrieval Research Collection, new organizations must first fill in a data release application form. Please contact Saadia Malik to obtain the form.