Today's content is increasingly a mixture of text, multimedia, and metadata. One way to format this mixed content is according to the adopted W3C standard for information repositories and exchanges, the so-called eXtensible Markup Language (XML).

This increasing use of XML, especially in scientific data repositories, Digital Libraries and on the Web, brought about an explosion in the development of XML systems, and in particular systems to store and access XML content. Whereas many of today's systems still treat documents as single large (text) blocks, XML offers the opportunity to exploit the logical structure of documents, which is explicitly represented by the XML markup, in order to allow for more precise access by giving more specific answers. Implementing this, more focused, retrieval paradigm means that an XML retrieval system needs not only to find relevant information in the XML documents, but also determine the appropriate level of granularity to return to the user. In addition, the relevance of a retrieved component is dependent on meeting both content and structural conditions.

Evaluating the effectiveness of XML retrieval systems, hence, requires a test collection where the relevance assessments are provided according to a relevance criterion, which takes into account the imposed structural aspects. A test collection as such has been built as a result of three rounds of the Initiative for the Evaluation of XML Retrieval (INEX 2002, INEX 2003 and INEX 2004). This initiative provides an opportunity for participants to evaluate their XML retrieval methods using uniform scoring procedures and a forum for participating organisations to compare their results. As part of a large-scale effort to improve the efficiency of research in information retrieval and digital libraries, this project initiated an international, coordinated effort to promote evaluation procedures for content-oriented XML retrieval.

In INEX 2005, participating organisations will be able to compare the retrieval effectiveness of their XML document retrieval systems and will contribute to the continuous construction of a large XML test collection. The test collection will also provide participants a means for future comparative and quantitative experiments. Due to copyright issues, only participating organisations will have access to the constructed test collection.

INEX test collection

The test collection consists of a set of XML documents, topics and relevance assessments. The topics and the relevance judgments are obtained through a collaborative effort from the participants. Detailed guidelines on the on-line topic submission, retrieval result submission, relevance assessment task, and evaluation metrics will be provided by INEX.

Documents

The INEX document collection is so far made up of the full-texts, marked up in XML, of 12,107 articles of the IEEE Computer Society's publications from 12 magazines and 6 transactions, covering the period of 1995-2002, and totalling 494 megabytes in size. The collection has a suitably complex XML structure (192 different content models in DTD) and contains scientific articles of varying length. On average an article contains 1,532 XML nodes, where the average depth of a node is 6.9. In INEX 2005, it is expected that additional documents from the IEEE Computer Society's publications will be added to the current document collection.

Topics

Each participating group will be asked to create a set of candidate topics, which are representative of the range of real user needs over the XML collection. The queries may be content-only (CO) or content-and-structure (CAS) queries, and broad or narrow topic queries. CO queries are free text queries, like those used in TREC, for which the retrieval system should retrieve relevant XML elements of varying granularity, while CAS queries contain explicit structural constraints, such as containment conditions. From the pooled set of candidate topics INEX will select a final set of topics to form part of the INEX test collection

Tasks

The general task, to be performed with the data and the final set of topics, will be the ad-hoc retrieval of XML documents. Similarly to information retrieval, we regard ad-hoc retrieval as a simulation of how a library might be used, where a static set of documents is searched using a new set of queries (topics). The main differences are that, in INEX, the library consists of XML documents, the queries may contain both content and structural conditions and, in response to a query, arbitrary XML elements may be retrieved from the library. Participants will be able to submit up to a fixed number of runs, each containing the top 1500 retrieval results for each of the selected topics.

In addition to the main general ad-hoc retrieval task, INEX 2005 will have the following two specific tasks, which continue from INEX 2004:

1. Relevance feedback task
2. Natural query language task

INEX 2005 will continue with the following two tracks that started in INEX 2004:

1. Heterogeneous collection track
2. Interactive track

Two additional tracks are planned for INEX 2005:

1. Document mining track
2. Multimedia track

Relevance assessments

Relevance assessments will be provided by the participating groups using INEX's on-line assessment system. Each assessor will judge around 2 topics, either the topics that they originally created or if these were removed from the final set of topics, then topics that were similar to their original queries or within their expertise. Please note that assessments take about one person week per topic! Participating groups will gain access to the completed INEX test collection only after they have completed their assessment task. .

Evaluation

The evaluation of the retrieval effectiveness of the XML retrieval engines used by the participants will be based on the constructed INEX test collection and uniform scoring techniques which will take into account the structural nature of XML documents, and overlap of answers.

Workshop and proceedings

Participants will be able to present their approaches and final results at the INEX 2005 workshop to be held in November in Dagstuhl. All descriptions of the approaches and results will be published in the INEX workshop pre-proceedings and the Web. Revised papers will be published in the INEX post-workshop final proceedings. As for INEX 2004, we are expected the INEX final proceedings to be published in the Springer's Lecture Notes in Computer Science (LNCS) series.

Data Handling Agreement

• INEX Collection

In order to have access to the data designated as the IEEE Computer Society XML Retrieval Research Collection, organizations (who did not sign the agreement in 2004) must first fill in a data release Application Form. The signed form must be sent (by express mail) to Saadia Malik at the address given in the Application Form (only the original copies of the forms are accepted, no electronic or fax versions). On receipt of the forms, you will be sent information on how to download the data.

Access to the data by an individual person is to be controlled by that person's organisation. The organisation may only grant access to people working under its control, i.e. its own members, consultants to the organisation, or individuals providing service to the organisation. All application forms by individuals to access the data must be signed by a person authorised by your organisation for such signatures. The individuals form must be kept by the organisation for any persons being involved at its site.

• Lonely Planet Collection

Application by an organisation to use the Lonely Planet Data.