Automatically Configuring Parallelism for Hybrid Layouts

Rana Faisal Munir; Alberto Abelló; Oscar Romero; Maik Thiele; Wolfgang Lehner

doi:10.1007/978-3-030-30278-8_15

Automatically Configuring Parallelism for Hybrid Layouts

Rana Faisal Munir^*
, Alberto Abelló
, Oscar Romero
, Maik Thiele
, Wolfgang Lehner

^*Corresponding author for this work

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

1 Citation (Scopus)

Abstract

Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.

Original language	English
Title of host publication	New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings
Editors	Tatjana Welzer, Vili Podgorelec, Aida Kamišalic Latific, Johann Eder, Robert Wrembel, Mikolaj Morzy, Mirjana Ivanovic, Johann Gamper, Theodoros Tzouramanis, Jérôme Darmont
Number of pages	6
Publisher	Physica-Verlag
Publication date	2019
Pages	120-125
ISBN (Print)	9783030302771
DOIs	https://doi.org/10.1007/978-3-030-30278-8_15
Publication status	Published - 2019
Externally published	Yes
Event	Workshops and doctoral consortium papers of the 23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019 - Bled, Slovenia Duration: 8 Sept 2019 → 11 Sept 2019

Conference

Conference	Workshops and doctoral consortium papers of the 23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019
Country/Territory	Slovenia
City	Bled
Period	08/09/2019 → 11/09/2019

Series	Communications in Computer and Information Science
Volume	1064
ISSN	1865-0929

Bibliographical note

Publisher Copyright:
© Springer Nature Switzerland AG 2019.

Keywords

Big data
Hybrid storage layouts
Parallelism
Parquet
Spark

Access to Document

10.1007/978-3-030-30278-8_15

AUB Link

Search for the material in Aalborg University Library's search engine

Cite this

Munir, R. F., Abelló, A., Romero, O., Thiele, M., & Lehner, W. (2019). Automatically Configuring Parallelism for Hybrid Layouts. In T. Welzer, V. Podgorelec, A. Kamišalic Latific, J. Eder, R. Wrembel, M. Morzy, M. Ivanovic, J. Gamper, T. Tzouramanis, & J. Darmont (Eds.), New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings (pp. 120-125). Physica-Verlag. https://doi.org/10.1007/978-3-030-30278-8_15

Munir, Rana Faisal ; Abelló, Alberto ; Romero, Oscar et al. / Automatically Configuring Parallelism for Hybrid Layouts. New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings. editor / Tatjana Welzer ; Vili Podgorelec ; Aida Kamišalic Latific ; Johann Eder ; Robert Wrembel ; Mikolaj Morzy ; Mirjana Ivanovic ; Johann Gamper ; Theodoros Tzouramanis ; Jérôme Darmont. Physica-Verlag, 2019. pp. 120-125 (Communications in Computer and Information Science, Vol. 1064).

@inproceedings{27e818b46b384838b5b745bfd75588f6,

title = "Automatically Configuring Parallelism for Hybrid Layouts",

abstract = "Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.",

keywords = "Big data, Hybrid storage layouts, Parallelism, Parquet, Spark",

author = "Munir, \{Rana Faisal\} and Alberto Abell{\'o} and Oscar Romero and Maik Thiele and Wolfgang Lehner",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2019.; Workshops and doctoral consortium papers of the 23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019 ; Conference date: 08-09-2019 Through 11-09-2019",

year = "2019",

doi = "10.1007/978-3-030-30278-8\_15",

language = "English",

isbn = "9783030302771",

series = "Communications in Computer and Information Science",

publisher = "Physica-Verlag",

pages = "120--125",

editor = "Tatjana Welzer and Vili Podgorelec and \{Kami{\v s}alic Latific\}, Aida and Johann Eder and Robert Wrembel and Mikolaj Morzy and Mirjana Ivanovic and Johann Gamper and Theodoros Tzouramanis and J{\'e}r{\^o}me Darmont",

booktitle = "New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings",

}

Munir, RF, Abelló, A, Romero, O, Thiele, M & Lehner, W 2019, Automatically Configuring Parallelism for Hybrid Layouts. in T Welzer, V Podgorelec, A Kamišalic Latific, J Eder, R Wrembel, M Morzy, M Ivanovic, J Gamper, T Tzouramanis & J Darmont (eds), New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings. Physica-Verlag, Communications in Computer and Information Science, vol. 1064, pp. 120-125, Workshops and doctoral consortium papers of the 23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019, Bled, Slovenia, 08/09/2019. https://doi.org/10.1007/978-3-030-30278-8_15

Automatically Configuring Parallelism for Hybrid Layouts. / Munir, Rana Faisal; Abelló, Alberto; Romero, Oscar et al.
New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings. ed. / Tatjana Welzer; Vili Podgorelec; Aida Kamišalic Latific; Johann Eder; Robert Wrembel; Mikolaj Morzy; Mirjana Ivanovic; Johann Gamper; Theodoros Tzouramanis; Jérôme Darmont. Physica-Verlag, 2019. p. 120-125 (Communications in Computer and Information Science, Vol. 1064).

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

TY - GEN

T1 - Automatically Configuring Parallelism for Hybrid Layouts

AU - Munir, Rana Faisal

AU - Abelló, Alberto

AU - Romero, Oscar

AU - Thiele, Maik

AU - Lehner, Wolfgang

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2019.

PY - 2019

Y1 - 2019

N2 - Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.

AB - Distributed processing frameworks process data in parallel by dividing it into multiple partitions and each partition is processed in a separate task. The number of tasks is always created based on the total file size. However, this can lead to launch more tasks than needed in the case of hybrid layouts, because they help to read less data for certain operations (i.e., projection, selection). The over-provisioning of tasks may increase the job execution time and induce significant waste of computing resources. The latter due to the fact that each task introduces extra overhead (e.g., initialization, garbage collection, etc.). To allow a more efficient use of resources and reduce the job execution time, we propose a cost-based approach that decides the number of tasks based on the data being read. The proposed cost-model can be utilized in a multi-objective approach to decide both the number of tasks and number of machines for execution.

KW - Big data

KW - Hybrid storage layouts

KW - Parallelism

KW - Parquet

KW - Spark

UR - https://www.scopus.com/pages/publications/85072990842

U2 - 10.1007/978-3-030-30278-8_15

DO - 10.1007/978-3-030-30278-8_15

M3 - Article in proceeding

AN - SCOPUS:85072990842

SN - 9783030302771

T3 - Communications in Computer and Information Science

SP - 120

EP - 125

BT - New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings

A2 - Welzer, Tatjana

A2 - Podgorelec, Vili

A2 - Kamišalic Latific, Aida

A2 - Eder, Johann

A2 - Wrembel, Robert

A2 - Morzy, Mikolaj

A2 - Ivanovic, Mirjana

A2 - Gamper, Johann

A2 - Tzouramanis, Theodoros

A2 - Darmont, Jérôme

PB - Physica-Verlag

T2 - Workshops and doctoral consortium papers of the 23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019

Y2 - 8 September 2019 through 11 September 2019

ER -

Munir RF, Abelló A, Romero O, Thiele M, Lehner W. Automatically Configuring Parallelism for Hybrid Layouts. In Welzer T, Podgorelec V, Kamišalic Latific A, Eder J, Wrembel R, Morzy M, Ivanovic M, Gamper J, Tzouramanis T, Darmont J, editors, New Trends in Databases and Information Systems - ADBIS 2019 Short Papers, Workshops BBIGAP, QAUCA, SemBDM, SIMPDA, M2P, MADEISD, and Doctoral Consortium 2019, Proceedings. Physica-Verlag. 2019. p. 120-125. (Communications in Computer and Information Science, Vol. 1064). doi: 10.1007/978-3-030-30278-8_15

Automatically Configuring Parallelism for Hybrid Layouts

Abstract

Conference

Bibliographical note

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

Cite this