D3.1 State-of-the-art report on data collection and analysis

Research output: Book/ReportReportResearchpeer-review

Abstract

Today, many countries aim to increase the share of energy consumed that comes from renewable
sources. Unfortunately, the electrical power produced from weather-dependent renewable energy
sources (RESs; e.g., wind turbines, solar panels) is produced in varying quantities that do not match
the varying energy needs. As more and more such renewable energy becomes available, it becomes
an increasingly difficult challenge to maintain an energy system that enables the effective use of all
available renewable energy. Consequently, tackling this problem is one of the top goals in the energy
domain.
The MIRACLE (Micro-Request-Based Aggregation, Forecasting and Scheduling of Energy Demand,
Supply and Distribution) project aims to invent and prototype key elements of an energy
system that is better able to accommodate large volume of electricity from renewable sources. The
approach, taken is based on micro-requests that allow an individual consumer/producer to specify
acceptable flexibilities in the amounts of energy consumed and the times when this is done.
The introduction of such micro-requests from millions of consumers/producers enables fine-grained
scheduling of consumption and production of electricity while maintaining a system-wide balance between
demand and supply. In order to appropriately manage very large volumes of micro-requests,
a reliable, distributed, and highly scalable computer system infrastructure is needed.
This deliverable concerns Work Package 3, “State-of-the-art report on data collection and analysis”
in the MIRACLE project. We first introduce MIRACLE’s application scenario along with the consequent
requirements to the data management infrastructure. Then we survey the state-of-the-art of
relevant, existing work on data collection, data integration, query processing, and query optimization
from the perspective of the projects requirements. Specifically, the survey covers the following key
topics: (1) virtually and materialized integrated systems, including column stores; (2) data exchange
solutions, including ETL tools, EAI servers, and data stream management systems; (3) web-scale
data management; (4) management of uncertainty in the context of probabilistic databases, OLAP,
and data streams; (5) management of multi-version data; (6) efficient tracking of continuous processes;
and (7) query optimization based on early aggregation and materialized views. Moreover,
relevant existing computer systems in the energy domain are covered. For all technologies surveyed,
the relation to MIRACLE is discussed.
Original languageEnglish
Publication statusPublished - 1 Jun 2010

Fingerprint

Computer systems
Agglomeration
Electricity
Data integration
Query processing
Wind turbines
Information management
Servers
Scheduling
Uncertainty

Cite this

@book{89a5672d877241afa9db88c293f78a8b,
title = "D3.1 State-of-the-art report on data collection and analysis",
abstract = "Today, many countries aim to increase the share of energy consumed that comes from renewable sources. Unfortunately, the electrical power produced from weather-dependent renewable energy sources (RESs; e.g., wind turbines, solar panels) is produced in varying quantities that do not match the varying energy needs. As more and more such renewable energy becomes available, it becomes an increasingly difficult challenge to maintain an energy system that enables the effective use of all available renewable energy. Consequently, tackling this problem is one of the top goals in the energy domain. The MIRACLE (Micro-Request-Based Aggregation, Forecasting and Scheduling of Energy Demand, Supply and Distribution) project aims to invent and prototype key elements of an energy system that is better able to accommodate large volume of electricity from renewable sources. The approach, taken is based on micro-requests that allow an individual consumer/producer to specify acceptable flexibilities in the amounts of energy consumed and the times when this is done. The introduction of such micro-requests from millions of consumers/producers enables fine-grained scheduling of consumption and production of electricity while maintaining a system-wide balance between demand and supply. In order to appropriately manage very large volumes of micro-requests, a reliable, distributed, and highly scalable computer system infrastructure is needed. This deliverable concerns Work Package 3, “State-of-the-art report on data collection and analysis” in the MIRACLE project. We first introduce MIRACLE’s application scenario along with the consequent requirements to the data management infrastructure. Then we survey the state-of-the-art of relevant, existing work on data collection, data integration, query processing, and query optimization from the perspective of the projects requirements. Specifically, the survey covers the following key topics: (1) virtually and materialized integrated systems, including column stores; (2) data exchange solutions, including ETL tools, EAI servers, and data stream management systems; (3) web-scale data management; (4) management of uncertainty in the context of probabilistic databases, OLAP, and data streams; (5) management of multi-version data; (6) efficient tracking of continuous processes; and (7) query optimization based on early aggregation and materialized views. Moreover, relevant existing computer systems in the energy domain are covered. For all technologies surveyed, the relation to MIRACLE is discussed.",
author = "Laurynas Siksnys and Matthias Boehm and Pedersen, {Torben Bach} and Jensen, {Christian S{\o}ndergaard} and Dalia Martisiut{\'e}",
year = "2010",
month = "6",
day = "1",
language = "English",

}

D3.1 State-of-the-art report on data collection and analysis. / Siksnys, Laurynas; Boehm, Matthias; Pedersen, Torben Bach; Jensen, Christian Søndergaard; Martisiuté, Dalia.

2010.

Research output: Book/ReportReportResearchpeer-review

TY - RPRT

T1 - D3.1 State-of-the-art report on data collection and analysis

AU - Siksnys, Laurynas

AU - Boehm, Matthias

AU - Pedersen, Torben Bach

AU - Jensen, Christian Søndergaard

AU - Martisiuté, Dalia

PY - 2010/6/1

Y1 - 2010/6/1

N2 - Today, many countries aim to increase the share of energy consumed that comes from renewable sources. Unfortunately, the electrical power produced from weather-dependent renewable energy sources (RESs; e.g., wind turbines, solar panels) is produced in varying quantities that do not match the varying energy needs. As more and more such renewable energy becomes available, it becomes an increasingly difficult challenge to maintain an energy system that enables the effective use of all available renewable energy. Consequently, tackling this problem is one of the top goals in the energy domain. The MIRACLE (Micro-Request-Based Aggregation, Forecasting and Scheduling of Energy Demand, Supply and Distribution) project aims to invent and prototype key elements of an energy system that is better able to accommodate large volume of electricity from renewable sources. The approach, taken is based on micro-requests that allow an individual consumer/producer to specify acceptable flexibilities in the amounts of energy consumed and the times when this is done. The introduction of such micro-requests from millions of consumers/producers enables fine-grained scheduling of consumption and production of electricity while maintaining a system-wide balance between demand and supply. In order to appropriately manage very large volumes of micro-requests, a reliable, distributed, and highly scalable computer system infrastructure is needed. This deliverable concerns Work Package 3, “State-of-the-art report on data collection and analysis” in the MIRACLE project. We first introduce MIRACLE’s application scenario along with the consequent requirements to the data management infrastructure. Then we survey the state-of-the-art of relevant, existing work on data collection, data integration, query processing, and query optimization from the perspective of the projects requirements. Specifically, the survey covers the following key topics: (1) virtually and materialized integrated systems, including column stores; (2) data exchange solutions, including ETL tools, EAI servers, and data stream management systems; (3) web-scale data management; (4) management of uncertainty in the context of probabilistic databases, OLAP, and data streams; (5) management of multi-version data; (6) efficient tracking of continuous processes; and (7) query optimization based on early aggregation and materialized views. Moreover, relevant existing computer systems in the energy domain are covered. For all technologies surveyed, the relation to MIRACLE is discussed.

AB - Today, many countries aim to increase the share of energy consumed that comes from renewable sources. Unfortunately, the electrical power produced from weather-dependent renewable energy sources (RESs; e.g., wind turbines, solar panels) is produced in varying quantities that do not match the varying energy needs. As more and more such renewable energy becomes available, it becomes an increasingly difficult challenge to maintain an energy system that enables the effective use of all available renewable energy. Consequently, tackling this problem is one of the top goals in the energy domain. The MIRACLE (Micro-Request-Based Aggregation, Forecasting and Scheduling of Energy Demand, Supply and Distribution) project aims to invent and prototype key elements of an energy system that is better able to accommodate large volume of electricity from renewable sources. The approach, taken is based on micro-requests that allow an individual consumer/producer to specify acceptable flexibilities in the amounts of energy consumed and the times when this is done. The introduction of such micro-requests from millions of consumers/producers enables fine-grained scheduling of consumption and production of electricity while maintaining a system-wide balance between demand and supply. In order to appropriately manage very large volumes of micro-requests, a reliable, distributed, and highly scalable computer system infrastructure is needed. This deliverable concerns Work Package 3, “State-of-the-art report on data collection and analysis” in the MIRACLE project. We first introduce MIRACLE’s application scenario along with the consequent requirements to the data management infrastructure. Then we survey the state-of-the-art of relevant, existing work on data collection, data integration, query processing, and query optimization from the perspective of the projects requirements. Specifically, the survey covers the following key topics: (1) virtually and materialized integrated systems, including column stores; (2) data exchange solutions, including ETL tools, EAI servers, and data stream management systems; (3) web-scale data management; (4) management of uncertainty in the context of probabilistic databases, OLAP, and data streams; (5) management of multi-version data; (6) efficient tracking of continuous processes; and (7) query optimization based on early aggregation and materialized views. Moreover, relevant existing computer systems in the energy domain are covered. For all technologies surveyed, the relation to MIRACLE is discussed.

M3 - Report

BT - D3.1 State-of-the-art report on data collection and analysis

ER -