David L. Hart

The eXtensible Resource Allocation Service (XRAS), a comprehensive allocations environment for managing the submission, review, and awarding of resource allocations has been essential to managing the resource allocation needs of the national cyberinfrastructure for the past decade. As a software-as-a-service platform, XRAS supports not only the needs of its primary stakeholder program, but also the processes of half a dozen other national and regional resource providers. Over the past decade, XRAS has continuously improved on its original feature set and added new features to better meet the needs of its clients. Today, XRAS supports a core workload of 500 allocation requests each quarter, and the development roadmap for the system is focused on expanding the types of resources XRAS supports, enabling resources to be integrated in novel ways, and increasing the number of resource federations that XRAS can support. In this paper, we describe key aspects of resource allocations that have guided XRAS development; discuss the current XRAS clients and our approach to sustainability; and briefly describe the XRAS architecture and how it integrates with client sites and the ORCID ecosystem. We then turn to the XRAS architecture and client integration capabilities and describe major features of XRAS developed over the past decade. Finally, we outline future work planned for XRAS.

This paper investigates the return on investment (ROI) in cyberinfrastructure (CI) facilities and services by comparing the value of end products created to the cost of operations. We assessed the cost of a US CI facility called XSEDE and the value of the end products created using this facility, categorizing end products according to the International Integrated Reporting Framework. The US federal government invested approximately $0.3B in operating the XSEDE ecosystem from 2016–2022. The estimated value of end products facilitated by XSEDE ranges from around $4.7B to $22.7B or more. Credit for the majority of these end products is shared among various contributors, including the XSEDE ecosystem. Granting the XSEDE ecosystem a seemingly reasonable percentage of credit for its contributions to end product creation suggests that the return on federal investment in the XSEDE ecosystem, in terms of value of end products created, was greater than one and possibly far greater than one. The Framework proved useful for addressing this question. Earlier work showed that the value of services provided by XSEDE was significantly greater than the cost of those services to the US federal government—a positive return on investment for delivery of services. Analyzing the financial efficiency of operations and the financial value of end products are two means for assessing the success of CI facilities in financial terms. Financial analyses should be used as one of many approaches for evaluating the success of CI facilities.

The Allocations Service for the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program is charged with accepting, reviewing, and processing researchers’ requests to use resources that are integrated into the ACCESS ecosystem. We present as a case study the metrics framework used to evaluate the Allocations Service project, a metrics framework that aligns with the project’s goals and identifies key performance indicators (KPIs). Several of our top-level KPIs reflect complex concepts and are composite measures built from suites of metrics compiled from two primary sources: a well-instrumented allocations and accounting system and an annual survey of the ACCESS researcher community. This approach allows us to describe and measure complex concepts such as “democratization” and “ecosystem access time” in a quantitative manner and to target improvements to project activities. The metrics framework is augmented by metrics to measure the performance of the project team, to describe general ecosystem and allocations activity, and to capture publications from the researcher community. We used this framework to gather and present data as part of the ACCESS Allocations Service first annual NSF panel review. The metrics were largely successful at communicating our progress, but we also encountered a few unexpected technical issues with the data and calculations themselves, which we are continuing to refine. Presented here as a case study, this approach to a metrics framework for the Allocations Service has proved valuable in complementing more subjective descriptions of the project, its accomplishments, and progress toward our goals.

This paper uses accounting concepts—particularly the concept of Return on Investment (ROI)—to reveal the quantitative value of scientific research pertaining to a major US cyberinfrastructure project (XSEDE—the eXtreme Science and Engineering Discovery Environment). XSEDE provides operational and support services for advanced information technology systems, cloud systems, and supercomputers supporting non-classified US research, with an average budget for XSEDE of US$20M+ per year over the period studied (2014–2021). To assess the financial effectiveness of these services, we calculated a proxy for ROI, and converted quantitative measures of XSEDE service delivery into financial values using costs for service from the US marketplace. We calculated two estimates of ROI: a Conservative Estimate, functioning as a lower bound and using publicly available data for a lower valuation of XSEDE services; and a Best Available Estimate, functioning as a more accurate estimate, but using some unpublished valuation data. Using the largest dataset assembled for analysis of ROI for a cyberinfrastructure project, we found a Conservative Estimate of ROI of 1.87, and a Best Available Estimate of ROI of 3.24. Through accounting methods, we show that XSEDE services offer excellent value to the US government, that the services offered uniquely by XSEDE (that is, not otherwise available for purchase) were the most valuable to the facilitation of US research activities, and that accounting-based concepts hold great value for understanding the mechanisms of scientific research generally.

Recent policy shifts on the part of funding agencies and journal publishers are causing changes in the acknowledgment and citation behaviors of scholars. A growing emphasis on open science and reproducibility is changing how authors cite and acknowledge “research infrastructures”—entities that are used as inputs to or as underlying foundations for scholarly research, including data sets, software packages, computational models, observational platforms, and computing facilities. At the same time, stakeholder interest in quantitative understanding of impact is spurring increased collection and analysis of metrics related to use of research infrastructures. This article reviews work spanning several decades on tracing and assessing the outcomes and impacts from these kinds of research infrastructures. We discuss how research infrastructures are identified and referenced by scholars in the research literature and how those references are being collected and analyzed for the purposes of evaluating impact. Synthesizing common features of a wide range of studies, we identify notable challenges that impede the analysis of impact metrics for research infrastructures and outline key open research questions that can guide future research and applications related to such metrics.

Each year, the Computational & Information Systems Laboratory (CISL) conducts a survey of its current and recent user community to gather a number of metrics about the scientific impact and outcomes from the use of CISL's high-performance computing systems, particularly peer-reviewed publications. However, with a modest response rate and reliance on self-reporting by users, the accuracy of the survey is uncertain as is the degree of that uncertainty. To quantify this uncertainty, CISL undertook a project that attempted to provide statistically supported limits on the accuracy and precision of the survey approach. We discovered limitations related to the range of users' HPC usage in our modeling phase, and several methods were attempted to adjust the model to fit the usage data. The resulting statistical models leverage data about the HPC usage associated with survey invitees to quantify the degree to which the survey undercounts the relevant publications. A qualitative assessment of the collected publications aligns with the statistical models, reiterates the challenges associated with acknowledgment for use of HPC resources, and suggests ways to improve the survey results further.

We present a framework that compares the publication impact based on a comprehensive peer analysis of papers produced by scientists using XSEDE and NCAR resources. The analysis is introducing a percentile ranking based approach of citations of the XSEDE and NCAR papers compared to peer publications in the same journal that do not use these resources. This analysis is unique in that it evaluates the impact of the two facilities by comparing the reported publications from them to their peers from within the same journal issue. From this analysis, we can see that papers that utilize XSEDE and NCAR resources are cited statistically significantly more often. Hence we find that reported publications indicate that XSEDE and NCAR resources exert a strong positive impact on scientific research.

Data storage needs continue to grow in most fields, and the cost per byte for tape remains lower than the cost for disk, making tape storage a good candidate for cost-effective long-term storage. However, the workloads suitable for tape archives differ from those for disk file systems, and archives must handle internally generated workloads that can be more demanding than those generated by end users (e.g., migration of data from an old tape technology to a new one). To better understand the variegated workloads, we have followed the first steps in the data science methodology. For anyone considering the use or deployment of a tape-based data archive or for anyone interested in details of data archives in the context of data science, this paper describes key aspects of data archive workloads.