Legal Adhoc Track

Overview

The purpose of this track is to evaluate adhoc search methods on a non-Web collection.

For this track the standard procedure is used.

Test Collection

Task Description for Participating Systems

Each participant is granted access to the collection and a set of queries. Queries are selected from logs of two search engines Kodeks and Park.ru. The total number of queries is about 13000.

Expected result is an ordered list of document URLs. Maximum list size is 100 per query.

Evaluation Methodology

• instructions for assessors:
  Assessors evaluate document relevance to the query basing on the extended description of the user information need.
• evaluation method: pooling (pool depth is 50)
• relevance scale:
  
  • yes / probably yes / perhaps yes / no / impossible to evaluate
• official metrics:
  
  • precision
  • recall
  • TREC 11-point precision/recall graph

Data Formats

• collection
• queries
• results