The First Russian Information Retrieval Evaluation Seminar
First seminar was organized in 2003 with final workshop attached to the Russian Conference on Digital Libraries (St. Petersburg, October 2003). We had nine applications for participation but only seven teams were able to complete tasks on schedule. Among the '03 participants were several important industry representatives including two major players on the Russian web search market. The participation from academia was lower probably because research prototypes were not ready for scale of considered tasks and deadlines were tight.
ROMIP'2003 had two tracks - "adhoc" retrieval and Web-site classification using 7Gb+ subset of the narod.ru domain.
Queries for "adhoc" track were selected from the daily log of the popular Russian Web retrieval system Yandex. To prevent fine-tuning of results participants were asked to perform 15000 queries and for each query submit the first 100 results to the organizing committee. Queries for evaluation (about 50) were selected after all the participants submit their results.
For the evaluation of results we used the TREC-like pooling mechanism. However our evaluation procedure had several significant differences:
The training set for the classification track was based on the existing Web catalog for narod.ru sites. We selected about 170 categories from the second level of hierarchy. Each of selected categories had at least 5 samples. Participants were asked to assign a list containing maximum 5 categories to each of 22000 web sites from the collection. At the evaluation stage all the assignments from 17 selected categories were judged by at least two assessors.
Contact us: romip[AT]romip.ru.