ETLMR: A Highly Scalable Dimensional ETL Framework Based on MapReduce

Research output: Contribution to journalConference article in JournalResearchpeer-review

24 Citations (Scopus)
1065 Downloads (Pure)

Abstract

Extract-Transform-Load (ETL) flows periodically populate data warehouses (DWs) with data from different source systems. An increasing challenge for ETL flows is processing huge volumes of data quickly. MapReduce is establishing itself as the de-facto standard for large-scale data-intensive processing. However, MapReduce lacks support for high-level ETL specific constructs, resulting in low ETL programmer productivity. This paper presents a scalable ETL framework, ETLMR, based on MapReduce. ETLMR has built-in native support for operations on DW-specific constructs such as star schemas, snowflake schemas and slowly changing dimensions (SCDs). This enables ETL developers to construct scalable MapReduce-based ETL flows with very few code lines. To achieve good performance and load balancing, a number of dimension and fact processing schemes are presented, including techniques for efficiently processing different types of dimensions. The paper describes the integration of ETLMR with a MapReduce framework and evaluates its performance on large realistic data sets. The experimental results show that ETLMR achieves very good scalability and compares favourably with other MapReduce data
warehousing tools.
Original languageEnglish
Book seriesLecture Notes in Computer Science
Volume6862
Pages (from-to)96-111
ISSN0302-9743
DOIs
Publication statusPublished - Sept 2011
Event13th International Conference on Data Warehousing and Knowledge Discovery - Toulouse, France
Duration: 29 Aug 20112 Sept 2011
Conference number: 13

Conference

Conference13th International Conference on Data Warehousing and Knowledge Discovery
Number13
Country/TerritoryFrance
CityToulouse
Period29/08/201102/09/2011

Fingerprint

Dive into the research topics of 'ETLMR: A Highly Scalable Dimensional ETL Framework Based on MapReduce'. Together they form a unique fingerprint.

Cite this