MAIME: A Maintenance Manager for ETL Processes

Darius Butkevicius, Philipp Daniel Freiberger, Frederik Madsen Halberg, Jacob Bach Hansen, Søren Jensen, Michael Tarp, Harry Xuegang Huang, Christian Thomsen

Publikation: Bidrag til bog/antologi/rapport/konference proceedingKonferenceartikel i proceedingForskningpeer review

2 Citationer (Scopus)
141 Downloads (Pure)

Abstract

The proliferation of business intelligence applications moves most organizations into an era where data becomes an essential part of the success factors. More and more business focus has thus been added to the integration and pro
cessing of data in the enterprise environment. Developing and maintaining Extraction-Transform-Load (ETL) processes becomes critical in most data-driven organizations. External Data Sources (EDSs) often change their schema which potentially leaves the ETL processes that extract data
from those EDSs invalid. Repairing these ETL processes is time-consuming and tedious. As a remedy, we propose MAIME as a tool to (semi-)automatically maintain ETL processes. MAIME works with SQL Server Integration Services (SSIS) and uses a graph model as a layer of abstraction
on top of SSIS Data Flow tasks (ETL processes). We introduce a graph alteration algorithm which propagates detected EDS schema changes through the graph. Modifications done to a graph are directly applied to the underlying ETL process. It can be configured how MAIME handles EDS schema changes for different SSIS transformations. For the considered set of transformations, MAIME can maintain SSIS Data Flow tasks (semi-)automatically. Compared to doing
this manually, the amount of user inputs is decreased by a factor of 9.5 and the spent time is reduced by a factor of 9.8 in an evaluation.
OriginalsprogEngelsk
TitelProceedings of the Workshops of the EDBT/ICDT 2017 Joint Conference (EDBT/ICDT 2017)
Antal sider10
ForlagCEUR Workshop Proceedings
Publikationsdato15 mar. 2017
Artikelnummer8
StatusUdgivet - 15 mar. 2017
BegivenhedNineteenth International Workshop On Design, Optimization, Languages and Analytical Processing of Big Data - Venice, Italien
Varighed: 21 mar. 201724 mar. 2017
Konferencens nummer: 19
http://www.info.univ-tours.fr/~marcel/dolap2017/

Workshop

WorkshopNineteenth International Workshop On Design, Optimization, Languages and Analytical Processing of Big Data
Nummer19
Land/OmrådeItalien
ByVenice
Periode21/03/201724/03/2017
Internetadresse
NavnCEUR Workshop Proceedings
Vol/bind1810
ISSN1613-0073

Fingeraftryk

Dyk ned i forskningsemnerne om 'MAIME: A Maintenance Manager for ETL Processes'. Sammen danner de et unikt fingeraftryk.

Citationsformater