Introduction

Structured document retrieval allows for the retrieval of document fragments, i.e. XML elements, containing relevant information. The main INEX Ad Hoc task focuses on text-based XML element retrieval. Although text is dominantly present in most XML document collections, other types of media can also be found in those collections. Existing research on multimedia information retrieval has already shown that it is far from trivial to determine the combined relevance of a document that contains several multimedia objects. The objective of the multimedia track is to exploit the XML structure that provides a logical level at which multimedia objects are connected, to improve the retrieval performance of an XML-driven multimedia information retrieval system.

The multimedia track will continue to provide an evaluation platform for the retrieval of multimedia document fragments. In addition, we want to create a discussion forum where the participating groups can exchange their ideas on different aspects of the multimedia XML retrieval task. Ideas raised here, may provide input for this the task descriptions for this year and/or the coming years.

Test Collections

• Wikipedia Ad Hoc XML collection:This is the same collection that is used for the Ad Hoc track. The INEX 2007 wikipedia collection is a copy of the INEX 2006 collection with image identifiers added to the <image> tags for those images that are part of the multimedia corpus. The assumption is that a user will be able to see images from the multimedia corpus (those with id > 0) in-place in the XML fragments when assessing a fragment. The identifiers refer to the images and metadata files in the wikipedia image XML collection.
• Wikipedia image XML collection: This XML collection is specially prepared for the Multimedia track. It consists of the metadata documents for the images in the Wikipedia collection. Each document in this collection contains exactly one image with information on the user who uploaded it, the copyright information, and often a short description. This corresponds to the information that is available on Wikipedia, consider for instance: http://en.wikipedia.org/wiki/Image:AnneFrankHouseAmsterdam.jpg.

Additional sources of information

• Image classification scores: For each image the classification scores for 101 different concepts are provided by the University of Amsterdam (UvA) (Examples of each concept). The UvA classifier is trained on manually annotated TRECVID video data and the concepts are picked for the broadcast news domain. The performance of these classifiers on the broad collection of Wikipedia images varies greatly, but we believe it may still be a useful source of information. For an impression of the quality of the classifications, have a look at the top 100 results for each concept. For making optimal use of this valuable source of information it may be possible to re-interpret some of the classes based on the new domain, for example, the concept anchor person may not be widely present in this collection, but the classifier results can be useful for finding persons, or perhaps the tennis classifier is very bad at finding tennis, but good at finding vertical lines. Also, it may be useful to first remove all non-photographic material (using the graphics concept?) since this seems to have confused the classifier often.

  Details of the classification techniques can be found in the following paper:
  C.G.M. Snoek, M. Worring, J.C. van Gemert, J.M. Geusebroek, and A.W.M. Smeulders. The Challenge Problem for Automated Detection of 101 Semantic Concepts in Multimedia.In Proceedings of ACM Multimedia, Santa Barbara, USA, October 2006.

  Download the full set of concept classifcations [192MB]

• Image features: A set of 120D feature vectors, one for each image, is available that has been used to derive the image classification scores. Participants can use these feature vectors to build a custom CBIR-system, without having to pre-process the image collection. The features are based on natural images statistics to compactly represent color invariant texture information by a Weibull distribution.

  Details of the classification techniques can be found in the following paper:
  Jan C. van Gemert, Jan-Mark Geusebroek, Cor J. Veenman, Cees G.M. Snoek, and Arnold W.M. Smeulders. Robust Scene Categorization by Learning Image Statistics in Context. In CVPR Workshop on Semantic Learning Applications in Multimedia, New York, USA, June 2006.

  Download the Features [95MB]

Task description

One tasks for the multimedia track is to retrieve relevant document fragments based on an information need with a (structured) multimedia character. A structured document retrieval approach in that case should be able to combine the relevance of the different media types into a single (meaningful) ranking that is presented to the user. Topic formats for this task will follow the Ad Hoc formats. This way, we can merge the topics in the set of Ad Hoc topics to get many submissions. Additional topic fields to indicate example images or concepts will be available for the multimedia topics. A second task that we plan to look at is pure image retrieval task. The goal here is to retrieve images rather than multimedia fragments. The target collection for this task is the Wikipedia image XML collection. Given an information need, participants are required to return a ranked list of documents (=image+metadata) from this collection. Both visual content and context of the images may be used for ranking.

Assessments and Evaluation Methodology

Schedule

Organisers