Towards a sustainable astronomical data infrastructure: Optimising linking data from the Rucio datalake to the users areas within the SKA Regional Centres Network
Open Access
- 9 January 2026
- journal article
- Published by CERN in Open Research Europe
- Vol. 6
Abstract
The distributed architecture of the SKA Regional Centre Network (SRCNet) aims to provide scientific communities worldwide with efficient computational and storage resources to exploit the massive data volumes produced by the SKA Observatory (SKAO). Given the amount of SKAO data, traditional data management paradigms — where data is transferred to computational resources— are no longer feasible. Instead, computational workflows must increasingly be relocated closer to data storage locations, emphasizing efficient data access strategies and avoiding unnecessary duplication or redundancy. In this context, we present PrepareData, a modular and extensible data delivery service developed within SRCNet prototyping activities. Our proposal for this service addresses the critical challenge of redundant data transfers and duplication at both node and user levels by enabling seamless delivery of requested datasets from local Rucio Storage Elements (RSEs) directly into users’ working environments. PrepareData operates as a local service within each SRCNet node and it is integrated into a broader ecosystem of federated services. Specifically, we designed and evaluated two distinct yet complementary implementations to avoid unnecessary data duplication and to enable a dynamic data bridge between the RSEs and the user storage areas, through: (1) a filesystem-based solution leveraging CephFS, which uses shared filesystem mount points and bind mounts to ensure consistent and immediate data availability of the data across computational nodes, and (2) a Kubernetes model using Persistent Volumes and Persistent Volume Claims, dynamically injecting data into a user's areas. To tackle this work we detail the architectural design and development, the technical implementation, the integration of both solutions with science enabling tools, such as JupyterHub, CARTA or virtually any application, and finally we provide a performance evaluation. This contribution provides a scalable and sustainable blueprint for data delivery in federated scientific infrastructures, supporting the broader goals of green computing and efficient resource utilisation.Keywords
Funding Information
- European Commission (824064)
- Spanish National Research Council (INFRA24023)
- Ministerio de Ciencia, Innovación y Universidades (TED2021-130231B-I00)
- Ministerio de Ciencia, Innovación y Universidades (PID2021-123930OB-C21)
- Ministerio de Ciencia, Innovación y Universidades (CEX2021-001131-S)
- Ministerio de Ciencia, Innovación y Universidades (PID2024-155817OB-I00)
This publication has 15 references indexed in Scilit:
- SKA Science Data Challenge 2: analysis and resultsMonthly Notices of the Royal Astronomical Society, 2023
- dCache: Inter-disciplinary storage systemEPJ Web of Conferences, 2021
- Square Kilometre Array Science Data Challenge 1: analysis and resultsMonthly Notices of the Royal Astronomical Society, 2020
- Big telescope, big data: towards exascale with the Square Kilometre ArrayPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2020
- Overview of the ATLAS distributed computing systemEPJ Web of Conferences, 2019
- Rucio: Scientific Data ManagementComputing and Software for Big Science, 2019
- EOS as the present and future solution for data storage at CERNJournal of Physics: Conference Series, 2015
- Deduplication-Based Energy Efficient Storage System in Cloud EnvironmentThe Computer Journal, 2014
- LOFAR: The LOw-Frequency ARrayAstronomy & Astrophysics, 2013
- dCache, Storage System for the FutureLecture Notes in Computer Science, 2006