ROMIP: Russian Information Retrieval Evaluation Seminar

 General principles 
 Test collections 
 Relevance tables 


Mixed Adhoc Track

Adhoc search in a mixed collection of documents


The purpose of this track is to evaluate adhoc search methods on a heterogeneous document collection.

For this track the standard procedure is used.

Test Collection

The source dataset is a union of two collections: the collection and the Legal documents collection.
Documents from all the archives narod.*, legal.* and *_training.* must be searched.

Task Description for Participating Systems

Each participant is granted access to the both document collections ( and legal documents collection) and a set of queries which is a union of queries from the Web adhoc and legal adhoc tracks.

Expected result is an ordered list of document URLs. Maximum list size is 100 per query.

Evaluation Methodology

  • instructions for assessors:
    Assessors evaluate document relevance to the query basing on the extended description of the user information need.
  • evaluation method: pooling (pool depth is 50)
  • relevance scale:
    • yes / probably yes / perhaps yes / no / impossible to evaluate
    • yes / no / impossible to evaluate
  • official metrics:
    • precision
    • recall
    • TREC 11-point precision/recall graph

Data Formats