Jan Musiał

The Copernicus Program is the largest and most successful public space program globally. It provides continuous data across various spectral ranges, with an archive exceeding 84 petabytes and a daily growth of approximately 20 TB, both of which are expected to increase further. The openness of its data has contributed to the widespread use of Earth observation and the development of commercial products utilizing open data in Europe and worldwide. The entire archive, along with cloud-based data processing capabilities, is available free of charge through the Copernicus Data Space Ecosystem initiative and continues to evolve to meet global user standards. This paper presents the process of creating the STAC Copernicus Data Space Ecosystem catalog—the largest and most comprehensive STAC catalog in terms of metadata globally. It details the workflow, starting from the development of a metadata model for Sentinel data, through efficient indexing based on the original metadata files accompanying the products, to result validation and backend system ingestion (via database DSN). A particular highlight is that this entire process is executed using a single tool, eometadatatool, initially developed by DLR, further enhanced, and released as open-source software by the CloudFerro team. The eometadatatool facilitates metadata extraction from the original files accompanying Copernicus program products and others (e.g., Landsat, Copernicus Contributing Missions) using a CSV file containing the metadata name, the file in which it occurs, and the path to the key within the file. Since the CDSE repository operates as an S3 resource offering users free access, the tool supports product access via S3 resources by default, configurable through environment variables. All the above characterizes eometadatatool as the most powerful stactool (a high-level command-line tool and Python library for working with STAC) package available, providing both valid STAC items and a method for uploading them to the selected backend. The standard specification itself has been influenced by the CDSE catalog development process, which contributed to the evolution of the standard by introducing version 1.1 and updated extensions (storage, eo, proj) that better meet user needs. The paper discusses the most significant modifications, their impact on the catalog’s functionality, and outlines the main differences. Particular attention is given to performance optimization due to the substantial data volume and high update frequency. The study examines the configuration and performance testing (using Locust) of the frontend layer (stac-fastapi-pgstac) and backend (pgstac). The stac-fastapi-pgstac implementation was deployed on a scalable Kubernetes cluster and underwent a product hydration process (specific to managing JSON data in pgstac), leveraging Python's native capabilities for this task. The pgstac schema was deployed on a dedicated bare-metal server with a PostgreSQL database, utilizing master-worker replication enabled through appropriate pgstac configuration. Both software tools are open source, and the achieved optimal configurations are documented and will be presented in detail. The presented solution empowers the community to fully utilize the new catalog, leverage its functionalities, and access open tools that enable independent construction of STAC catalogs compliant with ESA and community recommendations.

Nitrogen dioxide (NO2 ) pollution is one of the most dangerous threats for public health. It is responsible for respiratory diseases, cardiovascular diseases, asthma, and many others. The objective of the presented study was to verify the potential of the Sentinel-5 Precursor Tropospheric NO2 Column Number Density (NO2 TVCD), provided by Copernicus Data Space Ecosystem, to support air pollution monitoring in Poland. The implementation of the objective was divided into three stages:to develop a model which estimates a ground-level NO 2 concentration; to find spatial representativeness area (SR area) referring to existing ground stations; to propose localization of new ground stations. Model estimating a ground-level NO2 concentration was developed in order to get spatial distribution of NO2 pollution over whole Poland. As a input data there was used NO2 TVCD provided by Copernicus Data Space Ecosystem as well as meteorological factors like air temperature, wind speed, planetary boundary layer height, pressure, atmospheric circulation type, wind direction, solar adiation flux from ERA-5 provided by European Centre for Medium-Range Weather Forecasts (ECMWF) and anthropogenic conditions (nightlight intensity, population, roads density). Due to the need for high computing power and a constantly working environment, CREODIAS resources were used. There were used several machine learning approach among which random forest had been found as a most accurate. The results revealed that model demonstrated MAE values of 3.4 μg/m3 (MAPE~37%) and 3.2 μg/m3 (MAPE~31%) for the hourly and weekly estimates, respectively.Obtaining NO2 ground concentration for whole Poland allowed for investigation of spatial autocorrelation of the air pollution phenomenon and determination of SR area. There were used five methods:Global and Local Moran’s: it was found strong spatial autocorrelation (Global Moran’s=0.99 and p-value

The Copernicus Data Space Ecosystem provides open access to the petabyte-scale EO data repository and to a wide range of tools and services, limited to some predefined quoatas. For users who would like to develop commercial services or for those who would like to have larger quotas/unlimited access to services the offer of CREODIAS platform is the solution. In this study an example of such a (pre)commercial service will be presented which publishes the Copernicus Sentinel-1 and Sentinel-2 products in the form of a Web Map Service (WMS) and WCS (Web Coverage Service). The architecture of the services based on the Kubernetes cluster allows horizontal scaling of a service along with a number of users requests. The WMS/WCS services to be presented combine data discovery, access, (pre)-processing, publishing (rendering) and dissemination capabilities available within a single RESTful (Representational state transfer) query. This gives a user great flexibility in terms of on-the-fly data extraction across a specific AOI (Area Of Interest), mosaicing, reprojection, simple band processing (cloud masking, normalized difference vegetation), rendering. The performance of the Copernicus Data Space Ecosystem and CREODIAS platform combined with the efficient software (Postgres 15 with PostGIS extension, MapServer with GDAL backend and Kakadu JPEG2000 driver) allows to achieve WMS/WCS service response time below 1 second on average. This in turn, gives a potential for massive parallelization of the computations given the horizontal scaling of the Kubernetes cluster.

Identification of crops over large areas is necessary for monitoring agricultural production, establishing food security policies, or controlling compliance with the Common Agricultural Policy. Data acquired by Sentinel-1 and Sentinel-2 satellite constellations with a combination of high temporal and spatial resolutions has already demonstrated, separately or fused, its ability to classify crops. However, here we propose a methodology for crop type classification based on Sentinel-2 data focusing on a few specific aspects that have been often not addressed simultaneously in the previous works:Comparison of processing steps when single field representation is generated, i.e. multiple-pixel representation (Pixel-Set Encoder, https://doi.org/10.48550/arXiv.1911.07757) vs zonal statistics Application of various machine and deep learning classifiers, from Random Forest as a background, through Long Short-Term Memory and Gated Recurrent Unit, to novel Transformers Providing reliable full probability distribution of all crops for each classified crop field that allow for flexible user-specific minimalization of omission or commission errors Applicability in operational mode on a large beyond-country-scale i.e. for tens of millions of crop fields Efficient use of EO-data open cloud infrastructures such us the Copernicus Data Space Ecosystem or CREODIAS The methodology has been already implemented and validated for Austria and Poland using Land Parcel Information System as a reference, but the list of countries will continue to grow. For Poland, 41 crops were successfully classified in 2022, for which the precision ranged from 0. 64 (pepper) to 0.97 (maize, beetroot, winter rape), and the overall classification accuracy (kappa) was 0.88. However, using a threshold of 0.85 on the probabilities allowed kappa of 0.95 and a range of accuracy (precision) for crops from 0.88 to 1.00, with 26 crops classified with an accuracy above 0.95. For Austria, for the years 2018-2021, a kappa coefficient of 0.85 to 0.9 was obtained without applying thresholds on probabilities. For Poland, classification was also carried out during the growing season reaching kappa of: 0.78 (in May), 0.84 (in June), 0.87 (in July) and 0.88 at the end of the season.With this approach, we also demonstrate how to use the Copernicus Data Space Ecosystem to efficiently extract information from big volumes of satellite data - here from Sentinel-2 and tens of millions of agricultural parcels.

The Copernicus Data Space Ecosystem (CDSE), a pivotal initiative of the European Space Agency (ESA) and funded by the European Commission, represents a groundbreaking advancement in Earth Observation (EO). It is an integral part of the ambitious Copernicus Programme, which has revolutionized EO data access and utilization across various sectors. The CDSE is more than a mere data repository; it is a collaborative platform where knowledge, expertise, and resources are shared and enhanced. This ecosystem approach is transforming the EO landscape, encouraging a more inclusive and participatory model where researchers, policymakers, businesses, and citizen scientists contribute to and benefit from the wealth of EO data. The CDSE stands as a beacon of international cooperation and a driver for sustainable development, setting new standards for leveraging EO data in addressing some of the most pressing global challenges.Central to the CDSE is its commitment to federation and cooperation, offering a flexible framework for integration and participation to data providers, remote sensing experts, and application developers. This approach fosters innovation and collaboration, ensuring that the ecosystem benefits from a diverse range of contributions. Data providers can upload their data to the ecosystem’s object storage, making it available to CDSE users under free or commercial conditions. This not only enhances the CDSE’s data repository but also expands the providers' reach. Remote sensing experts contribute their algorithms and workflows to the openEO Algorithm Plaza, a collaborative space for sharing and enhancing EO technologies and applications. Application developers can utilize CDSE's infrastructure and data to create solutions for specific needs like agricultural monitoring, demonstrating the ecosystem’s potential for application development.The CDSE's principle of federation extends beyond individual contributors to collaborations with other platform providers, enhancing the range of services and capabilities available to users. These partnerships are crucial for the ecosystem's growth, ensuring access to the best tools and data. The governance of the Data Space Ecosystem incorporates several third-party commercial providers and is designed to grow with additional contributions. It is based on European values of data sovereignty and privacy, ensuring trust, collaboration, and sustainability. The ecosystem maintains a clear distinction between publicly funded services and third-party offerings, both free and commercial. This open and federated approach makes CDSE a comprehensive solution for EO needs and solidifies its position as a leader in the EO community.

The Copernicus Data Space Ecosystem introduces an innovative approach to accessing large Earth Observation (EO) data through streaming services and APIs. This study elaborates on advantages of the novel data discovery, streaming and access APIs within the Data Space, emphasizing key components such as: STAC catalog, OData and OpenSearch querying APIs, Browser, S3 interface, Jupyter Hub, openEO, and Sentinel Hub’s APIs.The OData (Open Data Protocol) stands as the ISO/IEC approved standard, outlining best practices for designing and utilizing REST APIs. It harmonizes various elements such as request and response headers, status codes, HTTP methods, URL conventions, media types, payload formats, and query options. The OpenSearch catalogue enables the search for Copernicus data through a standardized web service. For technical details, the OpenSearch specification serves as a reference. The results from this web service are delivered as GeoJSON feature collections, with each feature representing an Earth observation 'product' and referencing the location of the actual data.STAC (Spatio-Temporal Asset Catalog) emerges as a cutting-edge EO data management standard with robust data discovery and access capabilities. Built on simple JSON entries with resource-pointing URLs, it boasts flexibility in the types of data stored and accommodates new extensions and capabilities seamlessly. The hierarchical structure includes collections (e.g., Sentinel-2 L2A), subdivided into items representing satellite products. These items further break down into assets corresponding to spectral bands. This detailed STAC catalogue facilitates user searches for specific spectral bands and enables streaming of spatial data subsets using HTTP range requests.JupyterHub is a central platform for managing Jupyter Notebooks, simplifying tasks like resource allocation and user authentication. Copernicus Data Space Ecosystem users get free access, but with limited resources. Jupyter Notebooks offer a user-friendly interface for EO data prototyping, integrating with Python SDKs like Sentinel Hub and OpenEO. They grant direct access to the EO data repository, eliminating dependency hassles with pre-configured kernels for immediate prototyping. Sentinel Hub, a satellite imagery processing service, that excels in on-the-fly actions like gridding, re-projection, mosaicking, etc. It efficiently fetches data for web applications or analytical processes like ML, based on satellite data without replication or pre-processing. Several APIs, including OGC services, contribute to the Data Space Browser for advanced data visualization. Key APIs within the Copernicus Data Space Ecosystem include: BYOC (Bring Your Own COG) API, (Batch) Processing API, (Batch) Statistical API. These APIs empower users to perform diverse tasks, from data ingestion to advanced statistical analyses.OpenEO, revolutionizes geospatial data processing and analysis by offering a unified, interoperable platform for developers, researchers, and scientists. This framework empowers users to address complex geospatial challenges collaboratively, leveraging distributed computing environments and cloud-based resources. The collaborative nature of openEO enables seamless sharing of code, workflows, and data processing methods across platforms, advancing the accessibility, scalability, and reproducibility of Earth Observation (EO) data. OpenEO's intuitive libraries in Python, R, and JavaScript simplify the analysis of diverse datasets. In addition to the SDK, openEO features the Web Editor - an online tool for visually constructing workflows.

The main objective of the presented study was to verify the potential of the Sentinel-5 Precursor (S-5P) Tropospheric NO₂ Column Number Density (NO₂ TVCD) to support air pollution monitoring in Poland. The secondary objective of this project was to establish a relationship between air pollution and meteorological conditions. The ERA-5 data together with the NO₂ TVCD product and auxiliary data were further assimilated into an artificial intelligence model in order to estimate surface NO₂ concentrations. The results revealed that the random forest method was the most accurate method for estimating the surface NO₂. The random forest model demonstrated MAE values of 3.4 μg/m³ (MAPE~37%) and 3.2 μg/m³ (MAPE~31%) for the hourly and weekly estimates, respectively. It was observed that the proposed model could be used for at least 120 days per year due to the cloud-free conditions. Further, it was found that the S-5P NO₂ TVCD was the most important variable, which explained more than 50% of the predictions. Other important variables were the nightlights, solar radiation flux, road density, population, and planetary boundary layer height. The predictions obtained with the proposed model were better fitted to the actual surface NO₂ concentrations than the CAMS median ensemble estimations (~15% better accuracy).

Timely crop yield forecasts at a national level are substantial to support food policies, to assess agricultural production, and to subsidize regions affected by food shortage. This study presents an operational crop yield forecasting system for Poland that employs freely available satellite and agro-meteorological products provided by the Copernicus programme. The crop yield predictors consist of: (1) Vegetation condition indicators provided daily by Sentinel-3 OLCI (optical) and SLSTR (thermal) imagery, (2) a backward extension of Sentinel-3 data (before 2018) derived from cross-calibrated MODIS data, and (3) air temperature, total precipitation, surface radiation, and soil moisture derived from ERA-5 climate reanalysis generated by the European Centre for Medium-Range Weather Forecasts. The crop yield forecasting algorithm is based on thermal time (growing degree days derived from ERA-5 data) to better follow the crop development stage. The recursive feature elimination is used to derive an optimal set of predictors for each administrative unit, which are ultimately employed by the Extreme Gradient Boosting regressor to forecast yields using official yield statistics as a reference. According to intensive leave-one-year-out cross validation for the 2000–2019 period, the relative RMSE for voivodships (NUTS-2) are: 8% for winter wheat, and 13% for winter rapeseed and maize. Respectively, for municipalities (LAU) it equals 14% for winter wheat, 19% for winter rapeseed, and 27% for maize. The system is designed to be easily applicable in other regions and to be easily adaptable to cloud computing environments such as Data and Information Access Services (DIAS) or Amazon AWS, where data sets from the Copernicus programme are directly accessible.