The Danish National Energy Data Lake: Requirements, Technical Architecture, and Tool Selection

Renewable Energy Sources such as wind and solardo not emit CO2 but their production vary considerably de-pending on time and weather. Thus, it is important to use theflexibilityin device loads to shift energy consumption to followthe production. For example, an Electrical Vehicle (EV) canbe charged very flexibly between arriving home at 5PM andleaving again at 7AM. Utilizing all available energy flexibilityrequires applying machine learning and AI on massive amountsof Big Data from many different actors and devices, ranging fromprivate consumers, over companies, to energy network operators,and using this to create digital solutions to enable and exploitflexibility. The projectFlexible Energy Denmark (FED)is buildingthe foundation for this for the entire Danish society. Specifically,FED collects data from a number ofLiving Labs (LLs)inrepresentative real-life physical environments. The data is storedin the Danish National Energy Data Lake, called FED Data Lake(FEDDL) to enable efficient and advanced analysis. FEDDL isbuilt using only open source tools which can run both on-premiseand in cloud settings. In this paper, we describe the requirementsfor FEDDL based on a representative LL case study, present itstechnical architecture, and provide a comparison of relevant toolsalong with the arguments for which ones we selected.