D3.1 State-of-the-art report on data collection and analysis

Today, many countries aim to increase the share of energy consumed that comes from renewable
sources. Unfortunately, the electrical power produced from weather-dependent renewable energy
sources (RESs; e.g., wind turbines, solar panels) is produced in varying quantities that do not match
the varying energy needs. As more and more such renewable energy becomes available, it becomes
an increasingly difficult challenge to maintain an energy system that enables the effective use of all
available renewable energy. Consequently, tackling this problem is one of the top goals in the energy
domain.
The MIRACLE (Micro-Request-Based Aggregation, Forecasting and Scheduling of Energy Demand,
Supply and Distribution) project aims to invent and prototype key elements of an energy
system that is better able to accommodate large volume of electricity from renewable sources. The
approach, taken is based on micro-requests that allow an individual consumer/producer to specify
acceptable flexibilities in the amounts of energy consumed and the times when this is done.
The introduction of such micro-requests from millions of consumers/producers enables fine-grained
scheduling of consumption and production of electricity while maintaining a system-wide balance between
demand and supply. In order to appropriately manage very large volumes of micro-requests,
a reliable, distributed, and highly scalable computer system infrastructure is needed.
This deliverable concerns Work Package 3, “State-of-the-art report on data collection and analysis”
in the MIRACLE project. We first introduce MIRACLE’s application scenario along with the consequent
requirements to the data management infrastructure. Then we survey the state-of-the-art of
relevant, existing work on data collection, data integration, query processing, and query optimization
from the perspective of the projects requirements. Specifically, the survey covers the following key
topics: (1) virtually and materialized integrated systems, including column stores; (2) data exchange
solutions, including ETL tools, EAI servers, and data stream management systems; (3) web-scale
data management; (4) management of uncertainty in the context of probabilistic databases, OLAP,
and data streams; (5) management of multi-version data; (6) efficient tracking of continuous processes;
and (7) query optimization based on early aggregation and materialized views. Moreover,
relevant existing computer systems in the energy domain are covered. For all technologies surveyed,
the relation to MIRACLE is discussed.