Kraaler: A User-Perspective Web Crawler

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

Abstract

Adaption of technologies being used on the web is changing frequently, requiring applications that interact with the web to continuously change their ability to parse it. This has led most web crawlers to either inherent simplistic parsing capabilities, differentiating from web browsers, or use a web browser with high-level interactions that restricts observable information. We introduce Kraaler, an open source universal web crawler that uses the Chrome Debugging Protocol, enabling the use of the Blink browser engine for parsing, while obtaining protocol-level information. The crawler stores information in a database and on the file system and the implementation has been evaluated in a predictable environment to ensure correctness in the collected data. Additionally, it has been evaluated in a real-world scenario, demonstrating the impact of the parsing capabilities for data collection.
Original languageEnglish
Title of host publication2019 Network Traffic Measurement and Analysis Conference (TMA)
Number of pages8
PublisherIEEE
Publication date5 Aug 2019
ISBN (Print)978-3-903176-17-1
ISBN (Electronic)978-3-903176-17-1
DOIs
Publication statusPublished - 5 Aug 2019
Event2019 Network Traffic Measurement and Analysis Conference (TMA) - Paris , France
Duration: 19 Jun 201921 Jun 2019

Conference

Conference2019 Network Traffic Measurement and Analysis Conference (TMA)
CountryFrance
CityParis
Period19/06/201921/06/2019

Cite this

Panum, Thomas Kobber ; Hansen, René Rydhof ; Pedersen, Jens Myrup. / Kraaler: A User-Perspective Web Crawler. 2019 Network Traffic Measurement and Analysis Conference (TMA). IEEE, 2019.
@inproceedings{02e7c7cf787743699da21ca2bf93f18d,
title = "Kraaler: A User-Perspective Web Crawler",
abstract = "Adaption of technologies being used on the web is changing frequently, requiring applications that interact with the web to continuously change their ability to parse it. This has led most web crawlers to either inherent simplistic parsing capabilities, differentiating from web browsers, or use a web browser with high-level interactions that restricts observable information. We introduce Kraaler, an open source universal web crawler that uses the Chrome Debugging Protocol, enabling the use of the Blink browser engine for parsing, while obtaining protocol-level information. The crawler stores information in a database and on the file system and the implementation has been evaluated in a predictable environment to ensure correctness in the collected data. Additionally, it has been evaluated in a real-world scenario, demonstrating the impact of the parsing capabilities for data collection.",
author = "Panum, {Thomas Kobber} and Hansen, {Ren{\'e} Rydhof} and Pedersen, {Jens Myrup}",
year = "2019",
month = "8",
day = "5",
doi = "10.23919/TMA.2019.8784660",
language = "English",
isbn = "978-3-903176-17-1",
booktitle = "2019 Network Traffic Measurement and Analysis Conference (TMA)",
publisher = "IEEE",
address = "United States",

}

Panum, TK, Hansen, RR & Pedersen, JM 2019, Kraaler: A User-Perspective Web Crawler. in 2019 Network Traffic Measurement and Analysis Conference (TMA). IEEE, 2019 Network Traffic Measurement and Analysis Conference (TMA), Paris , France, 19/06/2019. https://doi.org/10.23919/TMA.2019.8784660

Kraaler: A User-Perspective Web Crawler. / Panum, Thomas Kobber; Hansen, René Rydhof; Pedersen, Jens Myrup.

2019 Network Traffic Measurement and Analysis Conference (TMA). IEEE, 2019.

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

TY - GEN

T1 - Kraaler: A User-Perspective Web Crawler

AU - Panum, Thomas Kobber

AU - Hansen, René Rydhof

AU - Pedersen, Jens Myrup

PY - 2019/8/5

Y1 - 2019/8/5

N2 - Adaption of technologies being used on the web is changing frequently, requiring applications that interact with the web to continuously change their ability to parse it. This has led most web crawlers to either inherent simplistic parsing capabilities, differentiating from web browsers, or use a web browser with high-level interactions that restricts observable information. We introduce Kraaler, an open source universal web crawler that uses the Chrome Debugging Protocol, enabling the use of the Blink browser engine for parsing, while obtaining protocol-level information. The crawler stores information in a database and on the file system and the implementation has been evaluated in a predictable environment to ensure correctness in the collected data. Additionally, it has been evaluated in a real-world scenario, demonstrating the impact of the parsing capabilities for data collection.

AB - Adaption of technologies being used on the web is changing frequently, requiring applications that interact with the web to continuously change their ability to parse it. This has led most web crawlers to either inherent simplistic parsing capabilities, differentiating from web browsers, or use a web browser with high-level interactions that restricts observable information. We introduce Kraaler, an open source universal web crawler that uses the Chrome Debugging Protocol, enabling the use of the Blink browser engine for parsing, while obtaining protocol-level information. The crawler stores information in a database and on the file system and the implementation has been evaluated in a predictable environment to ensure correctness in the collected data. Additionally, it has been evaluated in a real-world scenario, demonstrating the impact of the parsing capabilities for data collection.

U2 - 10.23919/TMA.2019.8784660

DO - 10.23919/TMA.2019.8784660

M3 - Article in proceeding

SN - 978-3-903176-17-1

BT - 2019 Network Traffic Measurement and Analysis Conference (TMA)

PB - IEEE

ER -

Panum TK, Hansen RR, Pedersen JM. Kraaler: A User-Perspective Web Crawler. In 2019 Network Traffic Measurement and Analysis Conference (TMA). IEEE. 2019 https://doi.org/10.23919/TMA.2019.8784660