Aim/Motivation

The main motivation for the track is twofold. First, to investigate the behaviour of users when interacting with elements of XML documents, and secondly to investigate and develop approaches for element retrieval which are effective in user-based environments.

In INEX 2004 we addressed the first issue. One of the major outcomes was the need to investigate methods that can be supportive during the search process based on features extracted from the XML formatting. Problems that might be solved using such methods include Overlapping components, i.e., components from the same document at different ranks in the hit list. In 2005 we attempted to exploit the XML structure to achieve a better and more comprehensible presentation of results, e.g., by hierarchical hit lists and highlighting document parts. For 2006 we plan to work on collecting data through interactive experiments that can advance XML retrieval, e.g., by providing information about the behaviour of element retrieval users. Such information is needed for setting up models of elements retrieval users, and has a wider application to element retrieval in general.

Document Collection

The collection of documents will be the same one used for the ad hoc retrieval task, i.e., in 2006 the Wikipedia collection. No further relevance assessments are required on behalf of the participating sites.

Searchers

Each participating site will be responsible for recruiting test persons to participate in the study as searchers. A minimum number of test persons will be defined so that the obtained results can be statistically meaningful.

IR System

As in the two previous years, we plan to make a baseline system available to participants. The system will provide a basic functionality which will be agreed upon with the participating sites, and may include features like implicit relevance feedback and bookmarking of elements. In 2006 we also hope to provide an improved testbed around the IR system to ease the burden of controlling the experiment. In addition, we strongly encourage research groups to develop their own systems and interfaces to study a broader range of questions, as was done successfully by one group in 2005.

Topics

A number of 2006 CO topics will be used in the track, modified to meet the requirements of the track so that they can function as simulated work task situations. In these, more context on the motives and background of the topic is provided, and attempts are made to generalised the scope of the topics. Thereby the test persons can better place themselves in a situation where they would be motivated to search for information related to the work tasks. The aim is to enable the test persons to formulate and reformulate their own queries as realistically as possible in the interaction with the IR system.

Searcher Tasks

The test persons will need to identify documents which are useful/relevant for completing the requirements specified in the simulated work task. They can either identify these documents explicitly (e.g. by marking down a relevance score for each document) or implicitly (e.g. by saving or bookmarking useful documents). It may also be interesting to compare relevance assessments obtained from test persons to those obtained from the INEX assessors. A time limit will be set for each simulated work task.

Data Collection

Data will be mainly collected through system logging and through questionnaires filled by searchers. Appropriate questionnaires will be designed and distributed by the track organisers. The system logging could consist of the queries issued, the components returned by the system, the components actually viewed, relevance assessments of these, any browsing behaviour, as well as time stamps for each act of interaction between the test person and the system. All the above will need to be discussed in detail among the track participants.

Note

Please note that participants in the track must register for the main INEX initiative, and that to have access to the test collection, in particular the relevance assessments, participants must perform the relevance assessment task.

Schedule

Organisers