Data federation strategies for ATLAS using XRootD
Open Access
- 11 June 2014
- conference paper
- Published by IOP Publishing in Journal of Physics: Conference Series
- Vol. 513 (4) , 042049
Abstract
In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job-brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.Keywords
This publication has 5 references indexed in Scilit:
- AGIS: The ATLAS Grid Information SystemJournal of Physics: Conference Series, 2012
- The ATLAS Distributed Data Management project: Past and FutureJournal of Physics: Conference Series, 2012
- Data management in EGEEJournal of Physics: Conference Series, 2010
- PanDA: distributed production and distributed analysis system for ATLASJournal of Physics: Conference Series, 2008
- Managing ATLAS data on a petabyte-scale with DQ2Journal of Physics: Conference Series, 2008