Abstract
Adaption of technologies being used on the web is changing frequently, requiring applications that interact with the web to continuously change their ability to parse it. This has led most web crawlers to either inherent simplistic parsing capabilities, differentiating from web browsers, or use a web browser with high-level interactions that restricts observable information. We introduce Kraaler, an open source universal web crawler that uses the Chrome Debugging Protocol, enabling the use of the Blink browser engine for parsing, while obtaining protocol-level information. The crawler stores information in a database and on the file system and the implementation has been evaluated in a predictable environment to ensure correctness in the collected data. Additionally, it has been evaluated in a real-world scenario, demonstrating the impact of the parsing capabilities for data collection.
| Original language | English |
|---|---|
| Title of host publication | 2019 Network Traffic Measurement and Analysis Conference (TMA) |
| Editors | Stefano Secci, Isabelle Chrisment, Marco Fiore, Lionel Tabourier, Keun-Woo Lim |
| Number of pages | 8 |
| Publisher | IEEE |
| Publication date | 5 Aug 2019 |
| Pages | 153-160 |
| Article number | 8784660 |
| ISBN (Print) | 978-3-903176-17-1 |
| ISBN (Electronic) | 978-3-903176-17-1 |
| DOIs | |
| Publication status | Published - 5 Aug 2019 |
| Event | 2019 Network Traffic Measurement and Analysis Conference (TMA) - Paris , France Duration: 19 Jun 2019 → 21 Jun 2019 |
Conference
| Conference | 2019 Network Traffic Measurement and Analysis Conference (TMA) |
|---|---|
| Country/Territory | France |
| City | Paris |
| Period | 19/06/2019 → 21/06/2019 |