Dimuthu Wannipurage

Scientists often utilize personal laptops and workstations for initial research stages and turn to high-performance computing (HPC) supercomputers for compute-intensive tasks. However, seamless transitions between these environments are vital for enhancing productivity and accelerating research progress. Our paper presents the Cybershuttle Notebook Gateway, an open-source framework crafted to streamline this transition, optimize resource utilization, and reduce time-to-science for researchers. Leveraging JupyterLab, the framework extends kernel mechanics for seamless provisioning and connection to remote HPC cluster kernels. We delve into its architecture, which separates user authentication, kernel provisioning, and remote file system access. Additionally, we highlight practical capabilities like analyzing network load and CPU usage across various computational workloads.

Software-as-a-service science gateways provide user interfaces and middleware for accessing scientific software deployed on remote high-performance computing resources and clusters. Selecting the resource to use for a particular job submission may be left to the user, who may need more information to make good choices when selecting from multiple options. To address this problem, we have designed and developed an extensible, scalable metascheduling system that can provide automated scheduling capabilities based on resource availability and other characteristics. We develop a system model based on queuing theory to guide our implementation and provide a basis for analysis. In particular, we derive an efficiency metric from these considerations. We implement the metascheduling system within the open-source Apache Airavata framework for science gateways as a supplemental service for guiding the job submission capabilities. We measure efficiency in representative scenarios, observing efficiencies of greater than 70% even in scenarios with high input rates and low job acceptance rates.

Metadata catalogs are essential for enabling researchers to find and access relevant datasets. However, existing metadata catalog solutions have limitations, such as being domain-specific or using document-oriented databases that limit their scalability and flexibility. To address these issues, we introduce the Airavata Data Catalog - a multi-tenanted schema-free metadata catalog service that supports multiple domain-specific metadata schemas and access control mechanisms. This paper describes our approach to modeling the relational attributes of data products and their access control while supporting a schema-free, document-oriented approach to storing and searching metadata. Our approach offers significant improvements over existing solutions and demonstrates the feasibility of a scalable, flexible metadata catalog for scientific datasets.

This paper introduces Cybershuttle, a new type of user-facing cyberinfrastructure that provides seamless access to a range of resources for researchers, enhancing their productivity. The Cybershuttle Research Environment is built on open source Apache Airavata software and uses a hybrid approach that integrates locally deployed agent programs with centrally hosted middleware. This enables end-to-end integration of computational science and engineering research across a range of resources, including users’ local resources, centralized university computing and data resources, computational clouds, and NSF-funded national-scale computing centers. To ensure a user-centered approach, we have designed the scientific user environments with the best user-centered design practices.

Jupyter Notebooks are an enormously popular tool for creating and narrating computational research projects. They also have enormous potential for creating reproducible scientific research artifacts. Capturing the complete state of a notebook has additional benefits; for instance, the notebook execution may be split between local and remote resources, where the latter may have more powerful processing capabilities or store large or access-limited data. There are several challenges for making notebooks fully reproducible when examined in detail. The notebook code must be replicated entirely, and the underlying Python runtime environments must be identical. More subtle problems arise in replicating referenced data, external library dependencies, and runtime variable states. This paper presents solutions to these problems using Juptyer's standard extension mechanisms to create an archivable system state for a running notebook. We show that the overhead for these additional mechanisms, which involve interacting with the underlying Linux kernel, does not introduce substantial execution time overheads, demonstrating the approach's feasibility.