Rob Quick

The FAIR Data Principles of findability, accessibility, interoperability, and reusability provide a roadmap to reusing data analysis findings and reproducibility of AI-based data analysis. However, the work done during this research project has identified an issue that impacts AI reproducibility before code and data interoperability can be considered. Namely, code reusability when attempting to recreate the hardware and system-level software or the “runtime environment.” While attempting to determine the metadata needed to FAIRly couple datasets with AI algorithms, the research team determined that the problem of recreating the runtime environment of published state-of-the-art algorithms from the website Papers with Code provided a hurdle that must be overcome before automated data-algorithm coupling can be considered. While containerization solutions such as Docker or Singularity are created to address the issue of inconsistent runtime environments, few AI algorithm developers have embraced publishing containers alongside their AI codes, opting for documenting software dependencies, which only tell part of the runtime story. Additionally, containers are software, and many issues affecting the recreation of runtime environments can also affect orchestrated container solutions. This work describes the process employed to survey 75 openly available AI algorithms, as recorded by Papers with Code, spanning the machine learning areas of computer vision, audio analysis, and natural language processing. It also makes a case that merely publishing the algorithm software repository and datasets used to benchmark the accuracy of the analysis is not enough to enable the reproducibility of results or reuse of AI algorithms. Finally, it identifies the gap in runtime environment reusability between code repositories like GitHub and commercial services like Hugging Face to focus future work. It proposes providing solutions like a container library, enhanced documentation, and other methods to allow reproducible and reusable research and a roadmap for continuing toward a review of enhancing AI-ready data.

New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will be easier to develop and maintain, that it remains available in the future on new platforms, that it meets new needs, and that it is as reusable as possible. This report discusses a virtual half-day workshop on "Software Sustainability and High Energy Physics" that aimed 1) to bring together experts from HEP as well as those from outside to share their experiences and practices, and 2) to articulate a vision that helps the Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP) to create a work plan to implement elements of software sustainability. Software sustainability practices could lead to new collaborations, including elements of HEP software being directly used outside the field, and, as has happened more frequently in recent years, to HEP developers contributing to software developed outside the field rather than reinventing it. A focus on and skills related to sustainable software will give HEP software developers an important skill that is essential to careers in the realm of software, inside or outside HEP. The report closes with recommendations to improve software sustainability in HEP, aimed at the HEP community via IRIS-HEP and the HEP Software Foundation (HSF).

Data science skills are rapidly becoming a necessity in modern science. In response to this need, institutions and organizations around the world are developing research data science curricula to teach the programming and computational skills that are needed to build and maintain data infrastructures and maximize the use of available data. To date, however, few of these courses have included an explicit ethics component, and developing such components can be challenging. This paper describes a novel approach to teaching data ethics on short courses developed for the CODATA-RDA Schools for Research Data Science. The ethics content of these schools is centred on the concept of open and responsible (data) science citizenship that draws on virtue ethics to promote ethics of practice. Despite having little formal teaching time, this concept of citizenship is made central to the course by distributing ethics content across technical modules. Ethics instruction consists of a wide range of techniques, including stand-alone lectures, group discussions and mini-exercises linked to technical modules. This multi-level approach enables students to develop an understanding both of “responsible and open (data) science citizenship”, and of how such responsibilities are implemented in daily research practices within their home environment. This approach successfully locates ethics within daily data science practice, and allows students to see how small actions build into larger ethical concerns. This emphasises that ethics are not something “removed from daily research” or the remit of data generators/end users, but rather are a vital concern for all data scientists.

The Open Science Grid (OSG) relies upon the network as a critical part of the distributed infrastructures it enables. In 2012, OSG added a new focus area in networking with a goal of becoming the primary source of network information for its members and collaborators. This includes gathering, organizing, and providing network metrics to guarantee effective network usage and prompt detection and resolution of any network issues, including connection failures, congestion, and traffic routing. In September of 2015, this service was deployed into the OSG production environment. We will report on the creation, implementation, testing, and deployment of the OSG Networking Service. Starting from organizing the deployment of perfSONAR toolkits within OSG and its partners, to the challenges of orchestrating regular testing between sites, to reliably gathering the resulting network metrics and making them available for users, virtual organizations, and higher level services, all aspects of implementation will be reviewed. In particular, several higher-level services were developed to bring the OSG network service to its full potential. These include a web-based mesh configuration system, which allows central scheduling and management of all the network tests performed by the instances; a set of probes to continually gather metrics from the remote instances and publish it to different sources; a central network datastore (esmond), which provides interfaces to access the network monitoring information in close to real time and historically (up to a year) giving the state of the tests; and a perfSONAR infrastructure monitor system, ensuring the current perfSONAR instances are correctly configured and operating as intended. We will also describe the challenges we encountered in ongoing operations of the network service and how we have evolved our procedures to address those challenges. Finally we will describe our plans for future extensions and improvements to the service.