AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data

Nguyen Thi Thao Ho, Huy Vo, Mai Vu, Torben Bach Pedersen

Research output: Contribution to journal › Journal article › Research › peer-review

89 Downloads (Pure)

Abstract

Recent development in computing, sensing and crowd-sourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called "Big Data" to inform research and the decision-making process are virtually endless. In general, analyses have to be done across multiple data sets in order to bring out the most value of Big Data. A first important step is to identify temporal correlations between data sets. Given the characteristics of Big Data in terms of volume and velocity, techniques that identify correlations not only need to be fast and scalable, but also need to help users in ordering the correlations across temporal scales so that they can focus on important relationships. In this paper, we present AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual information to identify correlations at multiple temporal scales in large time series. Discovered correlations are suggested to users in an order based on the strength of the relationships. Our method supports an adaptive streaming technique minimizing duplicated computation and is implemented on top of Apache Spark for scalability. We also provide a comprehensive evaluation on the effectiveness and the scalability of AMIC using both synthetic and real-world data set.
Original languageEnglish
JournalIEEE Transactions on Big Data
ISSN2332-7790
DOIs
Publication statusE-pub ahead of print - 29 Mar 2019

Fingerprint

Time series
Scalability
Electric sparks
Explosions
Decision making
Availability
Big data
Time series data
Mutual information

Cite this

Ho, Nguyen Thi Thao ; Vo, Huy ; Vu, Mai ; Pedersen, Torben Bach. / AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data. In: IEEE Transactions on Big Data. 2019.
@article{79a8a188239848f08d87650c77fa8884,
title = "AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data",
abstract = "Recent development in computing, sensing and crowd-sourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called {"}Big Data{"} to inform research and the decision-making process are virtually endless. In general, analyses have to be done across multiple data sets in order to bring out the most value of Big Data. A first important step is to identify temporal correlations between data sets. Given the characteristics of Big Data in terms of volume and velocity, techniques that identify correlations not only need to be fast and scalable, but also need to help users in ordering the correlations across temporal scales so that they can focus on important relationships. In this paper, we present AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual information to identify correlations at multiple temporal scales in large time series. Discovered correlations are suggested to users in an order based on the strength of the relationships. Our method supports an adaptive streaming technique minimizing duplicated computation and is implemented on top of Apache Spark for scalability. We also provide a comprehensive evaluation on the effectiveness and the scalability of AMIC using both synthetic and real-world data set.",
author = "Ho, {Nguyen Thi Thao} and Huy Vo and Mai Vu and Pedersen, {Torben Bach}",
year = "2019",
month = "3",
day = "29",
doi = "10.1109/TBDATA.2019.2907987",
language = "English",
journal = "IEEE Transactions on Big Data",
issn = "2332-7790",
publisher = "IEEE",

}

Ho, NTT, Vo, H, Vu, M & Pedersen, TB 2019, 'AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data', IEEE Transactions on Big Data. https://doi.org/10.1109/TBDATA.2019.2907987

AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data. / Ho, Nguyen Thi Thao; Vo, Huy ; Vu, Mai; Pedersen, Torben Bach.

In: IEEE Transactions on Big Data, 29.03.2019.

Research output: Contribution to journal › Journal article › Research › peer-review

TY - JOUR

T1 - AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data

AU - Ho, Nguyen Thi Thao

AU - Vo, Huy

AU - Vu, Mai

AU - Pedersen, Torben Bach

PY - 2019/3/29

Y1 - 2019/3/29

N2 - Recent development in computing, sensing and crowd-sourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called "Big Data" to inform research and the decision-making process are virtually endless. In general, analyses have to be done across multiple data sets in order to bring out the most value of Big Data. A first important step is to identify temporal correlations between data sets. Given the characteristics of Big Data in terms of volume and velocity, techniques that identify correlations not only need to be fast and scalable, but also need to help users in ordering the correlations across temporal scales so that they can focus on important relationships. In this paper, we present AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual information to identify correlations at multiple temporal scales in large time series. Discovered correlations are suggested to users in an order based on the strength of the relationships. Our method supports an adaptive streaming technique minimizing duplicated computation and is implemented on top of Apache Spark for scalability. We also provide a comprehensive evaluation on the effectiveness and the scalability of AMIC using both synthetic and real-world data set.

AB - Recent development in computing, sensing and crowd-sourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called "Big Data" to inform research and the decision-making process are virtually endless. In general, analyses have to be done across multiple data sets in order to bring out the most value of Big Data. A first important step is to identify temporal correlations between data sets. Given the characteristics of Big Data in terms of volume and velocity, techniques that identify correlations not only need to be fast and scalable, but also need to help users in ordering the correlations across temporal scales so that they can focus on important relationships. In this paper, we present AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual information to identify correlations at multiple temporal scales in large time series. Discovered correlations are suggested to users in an order based on the strength of the relationships. Our method supports an adaptive streaming technique minimizing duplicated computation and is implemented on top of Apache Spark for scalability. We also provide a comprehensive evaluation on the effectiveness and the scalability of AMIC using both synthetic and real-world data set.

U2 - 10.1109/TBDATA.2019.2907987

DO - 10.1109/TBDATA.2019.2907987

M3 - Journal article

JO - IEEE Transactions on Big Data

JF - IEEE Transactions on Big Data

SN - 2332-7790

ER -

Ho NTT, Vo H, Vu M, Pedersen TB. AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data. IEEE Transactions on Big Data. 2019 Mar 29. https://doi.org/10.1109/TBDATA.2019.2907987

Access to Document