ROMIP: Russian Information Retrieval Evaluation Seminar

 General principles 
 Test collections 
 Relevance tables 


Relevance tables used in ROMIP tracks

For calculation of the evaluation metrics we use relevance tables most of which are based on the marks put down by the assessors.

The are several different types of relevance tables:

  • AND - "strong relevance" (all assessors mark the document as "relevant" to the given query)
  • OR - "loose relevance" (at least one assessor marks the document as "relevant" to the given query)
  • SINGLE - each document/query pair was evaluated by exactly one assessor
  • IDEAL - comparison with a reference catalog (without assessors)

Relevance tables by track: