MapReduce-based Dimensional ETL Made Easy

Liu Xiufeng; Christian Thomsen; Torben Bach Pedersen

MapReduce-based Dimensional ETL Made Easy

Liu Xiufeng, Christian Thomsen, Torben Bach Pedersen

Research output: Contribution to journal › Conference article in Journal › Research › peer-review

Abstract

This paper demonstrates ETLMR, a novel dimensional Extract–Transform–Load (ETL) programming framework that uses MapReduce to achieve scalability. ETLMR has builtin native support of data warehouse (DW) speciﬁc constructs such as star schemas, snowﬂake schemas, and slowly changing dimensions (SCDs). This makes it possible to build MapReducebased dimensional ETL ﬂows very easily. The ETL process can be conﬁgured with only few lines of code. We will demonstrate the concrete steps in using ETLMR to load data into a (partly snowﬂaked) DW schema. This includes conﬁguration of data sources and targets, dimension processing schemes, fact processing, and employment. In addition, we also present the scalability on large data sets.

Original language	English
Journal	Proceedings of the VLDB Endowment
Volume	5
Issue number	12
Pages (from-to)	1882-1885
Number of pages	4
ISSN	2150-8097
Publication status	Published - Aug 2012
Event	International Conference on Very Large Data Bases - Istanbul, Turkey Duration: 27 Aug 2012 → 31 Aug 2012 Conference number: 38

Conference

Conference	International Conference on Very Large Data Bases
Number	38
Country/Territory	Turkey
City	Istanbul
Period	27/08/2012 → 31/08/2012

Access to Document

http://vldb.org/pvldb/vol5/p1882_xiufengliu_vldb2012.pdf

AUB Link

Search for the material in Aalborg University Library's search engine

Cite this

@inproceedings{ac24fc9fe061417b9adc382b56de56e1,

title = "MapReduce-based Dimensional ETL Made Easy",

abstract = "This paper demonstrates ETLMR, a novel dimensional Extract–Transform–Load (ETL) programming framework that uses MapReduce to achieve scalability. ETLMR has builtin native support of data warehouse (DW) speciﬁc constructs such as star schemas, snowﬂake schemas, and slowly changing dimensions (SCDs). This makes it possible to build MapReducebased dimensional ETL ﬂows very easily. The ETL process can be conﬁgured with only few lines of code. We will demonstrate the concrete steps in using ETLMR to load data into a (partly snowﬂaked) DW schema. This includes conﬁguration of data sources and targets, dimension processing schemes, fact processing, and employment. In addition, we also present the scalability on large data sets.",

author = "Liu Xiufeng and Christian Thomsen and Pedersen, {Torben Bach}",

year = "2012",

month = aug,

language = "English",

volume = "5",

pages = "1882--1885",

journal = "Proceedings of the VLDB Endowment",

issn = "2150-8097",

publisher = "VLDB Endowment",

number = "12",

note = "International Conference on Very Large Data Bases ; Conference date: 27-08-2012 Through 31-08-2012",

}

TY - GEN

T1 - MapReduce-based Dimensional ETL Made Easy

AU - Xiufeng, Liu

AU - Thomsen, Christian

AU - Pedersen, Torben Bach

N1 - Conference code: 38

PY - 2012/8

Y1 - 2012/8

N2 - This paper demonstrates ETLMR, a novel dimensional Extract–Transform–Load (ETL) programming framework that uses MapReduce to achieve scalability. ETLMR has builtin native support of data warehouse (DW) speciﬁc constructs such as star schemas, snowﬂake schemas, and slowly changing dimensions (SCDs). This makes it possible to build MapReducebased dimensional ETL ﬂows very easily. The ETL process can be conﬁgured with only few lines of code. We will demonstrate the concrete steps in using ETLMR to load data into a (partly snowﬂaked) DW schema. This includes conﬁguration of data sources and targets, dimension processing schemes, fact processing, and employment. In addition, we also present the scalability on large data sets.

AB - This paper demonstrates ETLMR, a novel dimensional Extract–Transform–Load (ETL) programming framework that uses MapReduce to achieve scalability. ETLMR has builtin native support of data warehouse (DW) speciﬁc constructs such as star schemas, snowﬂake schemas, and slowly changing dimensions (SCDs). This makes it possible to build MapReducebased dimensional ETL ﬂows very easily. The ETL process can be conﬁgured with only few lines of code. We will demonstrate the concrete steps in using ETLMR to load data into a (partly snowﬂaked) DW schema. This includes conﬁguration of data sources and targets, dimension processing schemes, fact processing, and employment. In addition, we also present the scalability on large data sets.

UR - http://www.scopus.com/inward/record.url?scp=84872952928&partnerID=8YFLogxK

M3 - Conference article in Journal

SN - 2150-8097

VL - 5

SP - 1882

EP - 1885

JO - Proceedings of the VLDB Endowment

JF - Proceedings of the VLDB Endowment

IS - 12

T2 - International Conference on Very Large Data Bases

Y2 - 27 August 2012 through 31 August 2012

ER -

MapReduce-based Dimensional ETL Made Easy

Abstract

Conference

Access to Document

AUB Link

Other files and links

Fingerprint

Cite this