The Scholarly Impact of CLEF 2010-2017: A Google Scholar Analysis of CLEF Proceedings and Working Notes

Research output: Contribution to book/anthology/report/conference proceedingBook chapterResearchpeer-review


This chapter assesses the scholarly impact of the CLEF evaluation campaign by performing a bibliometric analysis of the citations of the CLEF 2010–2017 papers collected through Google Scholar. The analysis extends an earlier 2013 study by Tsikrika et al. of the CLEF Proceedings for the period 2000–2009 and compares the impact of the first half of CLEF to the second. It also extends the analysis by including the CLEF Working notes, a less formal but important part of the CLEF oeuvre. Results show that, despite the different nature of the peer-reviewed CLEF Proceedings papers and the less formal and much more numerous Working note papers, both types of publications have high citation impact. In particular, overview papers from the various labs and tasks in CLEF attract large amounts of citations in both Proceedings and Working Notes. A significant proportion of the total number of citations appear to be from outside CLEF—there are simply not enough CLEF papers every year to explain that many citations. In conclusion, the analysis of the productivity and citation impact of CLEF in the period 2010–2017 shows that CLEF is a very strong and vibrant initiative that has managed a major change of format between 2009/2010 and continues to produce relevant research, datasets and tools.
Original languageEnglish
Title of host publicationInformation Retrieval Evaluation in a Changing World : Lessons Learned from 20 Years of CLEF
EditorsNicola Ferro, Carol Peters
Place of PublicationCham, Switzerland
Publication date2019
ISBN (Print)978-3-030-22947-4
ISBN (Electronic)978-3-030-22948-1
Publication statusPublished - 2019
SeriesThe Information Retrieval Series


  • Impact of Conferences
  • Bibliometrics
  • Citation Analysis
  • Google Scholar
  • CLEF
  • Conference and Labs of the Evaluation Forum


Dive into the research topics of 'The Scholarly Impact of CLEF 2010-2017: A Google Scholar Analysis of CLEF Proceedings and Working Notes'. Together they form a unique fingerprint.

Cite this