Evaluation in context

Jaap Kamps*, Mounia Lalmas, Birger Larsen


Publikation: Bidrag til bog/antologi/rapport/konference proceedingKonferenceartikel i proceedingForskningpeer review

3 Citationer (Scopus)


All search happens in a particular context - such as the particular collection of a digital library, its associated search tasks, and its associated users. Information retrieval researchers usually agree on the importance of context, but they rarely address the issue. In particular, evaluation in the Cranfield tradition requires abstracting away from individual differences between users. This paper investigates if we can bring some of this context into the Cranfield paradigm. Our approach is the following: we will attempt to record the "context" of the humans already in the loop - the topic authors/assessors - by designing targeted questionnaires. The questionnaire data becomes part of the evaluation test-suite as valuable data on the context of the search requests. We have experimented with this questionnaire approach during the evaluation campaign of the INitiative for the Evaluation of XML Retrieval (INEX). The results of this case study demonstrate the viability of the questionnaire approach as a means to capture context in evaluation. This can help explain and control some of the user or topic variation in the test collection. Moreover, it allows to break down the set of topics in various meaningful categories, e.g. those that suit a particular task scenario, and zoom in on the relative performance for such a group of topics.

TitelResearch and Advanced Technology for Digital Libraries - 13th European Conference, ECDL 2009, Proceedings
Antal sider13
Publikationsdato1 dec. 2009
ISBN (Trykt)3642043453, 9783642043451
StatusUdgivet - 1 dec. 2009
Begivenhed13th European Conference on Research and Advanced Technologies for Digital Libraries, ECDL 2009 - Corfu, Grækenland
Varighed: 27 sep. 20092 okt. 2009


Konference13th European Conference on Research and Advanced Technologies for Digital Libraries, ECDL 2009
SponsorAlpha Bank, The European Research Consortium for Informatics and Mathematics, SWETS Information Services, InterOptics, EBSCO
NavnLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vol/bind5714 LNCS


Dyk ned i forskningsemnerne om 'Evaluation in context'. Sammen danner de et unikt fingeraftryk.