Evaluation measures for relevance and credibility in ranked lists

Christina Lioma, Jakob Grue Simonsen, Birger Larsen

Publikation: Bidrag til bog/antologi/rapport/konference proceedingKonferenceartikel i proceedingForskningpeer review

27 Citationer (Scopus)

Abstract

Recent discussions on alternative facts, fake news, and post truth politics have motivated research on creating technologies that allow people not only to access information, but also to assess the credibility of the information presented to them by information retrieval systems. Whereas technology is in place for filtering information according to relevance and/or credibility [15], no single measure currently exists for evaluating the accuracy or precision (and more generally effectiveness) of both the relevance and the credibility of retrieved results. One obvious way of doing so is to measure relevance and credibility effectiveness separately, and then consolidate the two measures into one. There at least two problems with such an approach: (I) it is not certain that the same criteria are applied to the evaluation of both relevance and credibility (and applying different criteria introduces bias to the evaluation); (II) many more and richer measures exist for assessing relevance effectiveness than for assessing credibility effectiveness (hence risking further bias). Motivated by the above, we present two novel types of evaluation measures that are designed to measure the effectiveness of both relevance and credibility in ranked lists of retrieval results. Experimental evaluation on a small human-annotated dataset (that we make freely available to the research community) shows that our measures are expressive and intuitive in their interpretation.

OriginalsprogEngelsk
TitelICTIR 2017 - Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval
Antal sider8
ForlagAssociation for Computing Machinery (ACM)
Publikationsdato1 okt. 2017
Sider91-98
ISBN (Elektronisk)978-1-4503-4490-6
DOI
StatusUdgivet - 1 okt. 2017
Begivenhed7th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2017 - Amsterdam, Holland
Varighed: 1 okt. 20174 okt. 2017

Konference

Konference7th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2017
Land/OmrådeHolland
ByAmsterdam
Periode01/10/201704/10/2017
SponsorSpecial Interest Group on Information Retrieval (ACM SIGIR)

Fingeraftryk

Dyk ned i forskningsemnerne om 'Evaluation measures for relevance and credibility in ranked lists'. Sammen danner de et unikt fingeraftryk.

Citationsformater