An open compute and data federation as an alternative to monolithic infrastructures for big Earth data analytics
Open Access
- 3 July 2022
- journal article
- research article
- Published by Taylor & Francis in Big Earth Data
- Vol. 7 (3) , 812-830
Abstract
An adequate compute and storage infrastructure supporting the full exploitation of Copernicus and Earth Observation datasets is currently not available in Europe. This paper presents the cross-disciplinary open-source technologies being leveraged in the C-SCALE project to develop an open federation of compute and data providers as an alternative to monolithic infrastructures for processing and analysing Copernicus and Earth Observation data. Three critical aspects of the federation and the chosen technologies are elaborated upon: (1) federated data discovery, (2) federated access and (3) software distribution. With these technologies the open federation aims to provide homogenous access to resources, thereby enabling its users to generate meaningful results quickly and easily. This will be achieved by abstracting the complexity of infrastructure resource access provisioning and orchestration, including discovery of data across distributed archives, away from the end-users. Which is needed because end-users wish to focus on analysing ready-to-use data products and models rather than spending their time on the setup and maintenance of complex and heterogeneous IT infrastructures. The open federation will support processing and analysing the vast amounts of Copernicus and Earth Observation data that are critical for the implementation of the Destination Earth resp. Digital Twins vision for a high precision digital model of the Earth to model, monitor and simulate natural phenomena and related human activities.Keywords
Funding Information
- European Union’s Horizon 2020 Research and Innovation Programme (101017529)
This publication has 14 references indexed in Scilit:
- The openEO API–Harmonising the Use of Earth Observation Cloud Services Using Virtual Data Cube FunctionalitiesRemote Sensing, 2021
- An Overview of Platforms for Big Earth Observation Data Management and AnalysisRemote Sensing, 2020
- A Geodata Fabric for the 21st CenturyEos, 2019
- Big Earth data: disruptive changes in Earth observation data management and analysis?International Journal of Digital Earth, 2019
- Recommendations to Improve Downloads of Large Earth Observation DataData Science Journal, 2018
- Google Earth Engine: Planetary-scale geospatial analysis for everyoneRemote Sensing of Environment, 2017
- High-resolution mapping of global surface water and its long-term changesNature, 2016
- Big earth observation data analytics2016
- Earth's surface water change over the past 30 yearsNature Climate Change, 2016
- Climate Data Challenges in the 21st CenturyScience, 2011