A Longitudinal Analysis of Search Engine Index Size

Antal Van den Bosch, Toine Bogers, Maurice De Kunder

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

3 Citations (Scopus)

Abstract

One of the determining factors of the quality of Web search engines is the size of their index. In addition to its influence on search result quality, the size of the indexed Web can also tell us something about which parts of the WWW are directly accessible to the everyday user. We propose a novel method of estimating the size of a Web search engine’s index by extrapolating from document frequencies of words observed in a large static corpus of Web pages. In addition, we provide a unique longitudinal perspective on the size of Google and Bing’s indexes over a nine-year period, from March 2006 until January 2015. We find that index size estimates of these two search engines tend to vary dramatically over time, with Google generally possessing a larger index than Bing. This result raises doubts about the reliability of previous one-off estimates of the size of the indexed Web. We find that much if not all of this variability can be explained by changes in the indexing and ranking infrastructure of Google and Bing. This casts further doubt on whether Web search engines can be used reliably for cross-sectional webometric studies.
Original languageEnglish
Title of host publicationProceedings of ISSI 2015 Istanbul : 15th International Society of Scientometrics and Informetrics Conference
EditorsAlbert Ali Salah, Yasar Tonta, Alkim Almila Akdag Salah, Cassidy Sugimoto, Umut Al
Number of pages12
PublisherBoğaziçi University Printhouse
Publication dateJun 2015
Pages71-82
ISBN (Print)978-975-518-381-7
Publication statusPublished - Jun 2015
Event15th International Society of Scientometrics and Informetrics Conference (ISSI 2015) - Istanbul, Turkey
Duration: 29 Jun 20153 Jul 2015
http://www.issi2015.org/en/

Conference

Conference15th International Society of Scientometrics and Informetrics Conference (ISSI 2015)
CountryTurkey
CityIstanbul
Period29/06/201503/07/2015
Internet address

Keywords

  • webometrics
  • search engines
  • Google
  • Bing
  • longitudinal study
  • index size estimation
  • Surface Web
  • World Wide Web
  • indexing

Fingerprint Dive into the research topics of 'A Longitudinal Analysis of Search Engine Index Size'. Together they form a unique fingerprint.

  • Activities

    • 1 Conference organisation or participation

    15th International Society of Scientometrics and Informetrics Conference (ISSI 2015)

    Toine Bogers (Participant)

    29 Jun 20153 Jul 2015

    Activity: Attending an eventConference organisation or participation

    Cite this

    Van den Bosch, A., Bogers, T., & De Kunder, M. (2015). A Longitudinal Analysis of Search Engine Index Size. In A. A. Salah, Y. Tonta, A. A. A. Salah, C. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 Istanbul : 15th International Society of Scientometrics and Informetrics Conference (pp. 71-82). Boğaziçi University Printhouse. http://www.issi2015.org/files/downloads/all-papers/0071.pdf