D. Aaron Roberts

Whitepaper #342 in the Decadal Survey for Solar and Space Physics (Heliophysics) 2024-2033. Main topics: infrastructure/workforce/other programmatic. Additional topics: research tools and infrastructure.

The Heliophysics Application Programmer’s Interface (HAPI) is a standard mechanism for accessing time series data. Because of its adoption at multiple Heliophysics (HP) and Space Weather (SW) data centers, it is now a useful way to reach many different resources within the community. It is also a COSPAR standard. The use of HAPI so far has been as a standard layer on top of a traditional mission or instrument archive, where HAPI lives alongside an existing, custom web-based computer interface. We will give highlights of recent additions to the specification which is now at version 3.0, with 3.1 around the corner. We also will present explorations into two new ways in which HAPI can be utilized. 1) HAPI as a way to access output from model runs, which can generate large volumes of data at various time cadences and spatial distributions. 2) HAPI as an interface over cloud-based data resources. In the cloud, HAPI can connect large volumes of data to scientist-friendly front-end analysis capabilities, such as a JupyterHub or potentially a Pangeo-like environment. The evolution of HAPI and its uses is expected to keep enhancing interoperability among Heliophysics and Space Weather resources.

Heliophysics data analysis often involves combining diverse science measurements, many of them captured as time series. Although there are now only a few commonly used data file formats, the diversity in mechanisms for automated access to and aggregation of such data holdings can make analysis that requires intercomparison of data from multiple data providers difficult. The Heliophysics Application Programmer's Interface (HAPI) is a recently developed standard for accessing distributed time series data to increase interoperability. The HAPI specification is based on the common elements of existing data services, and it standardizes the two main parts of a data service: the request interface and the response data structures. The interface is based on the REpresentational State Transfer (REST) or RESTful architecture style, and the HAPI specification defines five required REST endpoints. Data are returned via a streaming format that hides file boundaries; the metadata is detailed enough for the content to be scientifically useful, e.g., plotted with appropriate axes layout, units, and labels. Multiple mature HAPI‐related open‐source projects offer server‐side implementation tools and client‐side libraries for reading HAPI data in multiple languages (IDL, Java, MATLAB, and Python). Multiple data providers in the US and Europe have added HAPI access alongside their existing interfaces. Based on this experience, data can be served via HAPI with little or no information loss compared to similar existing web interfaces. Finally, HAPI has been recommended as a COSPAR standard for time series data delivery.

Heliophysics data analysis often involves combining diverse science measurements, many of them captured as time series. Although there are now only a few commonly used data file formats, the diversity in mechanisms for automated access to and aggregation of such data holdings can make analysis that requires inter-comparison of data from multiple data providers difficult. The Heliophysics Application Programmer’s Interface (HAPI) is a recently developed standard for accessing distributed time-series data to increase interoperability. The HAPI specification is based on the common elements of existing data services, and it standardizes the two main parts of a data service: the request interface and the response data structures. The interface is based on the REpresentational State Transfer (REST) or RESTful architecture style, and the HAPI specification defines five required REST endpoints. Data are returned via a streaming format that hides file boundaries; the metadata is detailed enough for the content to be scientifically useful, e.g., plotted with appropriate axes layout, units, and labels. Multiple mature HAPI-related open-source projects offer server-side implementation tools and client-side libraries for reading HAPI data in multiple languages (IDL, Java, MATLAB, and Python). Multiple data providers in the US and Europe have added HAPI access alongside their existing interfaces. Based on this experience, data can be served via HAPI with little or no information loss compared to similar existing web interfaces. Finally, HAPI has been recommended as a COSPAR standard for time series data delivery.

Heliophysics data analysis often involves combining diverse science measurements, many of them captured as time series. Although there are now only a few commonly used data file formats, the diversity in mechanisms for automated access to and aggregation of such data holdings can make analysis that requires inter-comparison of data from multiple data providers difficult. The Heliophysics Application Programmer’s Interface (HAPI) is a recently developed standard for accessing distributed time-series data to increase interoperability. The HAPI specification is based on the common elements of existing data services and it standardizes the two main parts of a data service: the request interface and the response data structures. The interface is based on the REpresentational State Transfer (REST) or RESTful architecture style, and the HAPI specification defines five required REST endpoints. Data are returned via a streaming format that hides file boundaries; the metadata is detailed enough for the content to be scientifically useful, e.g., plotted with appropriate axes layout, units, and labels. Multiple mature HAPI-related open-source projects offer server-side implementation tools and client-side libraries for reading HAPI data in multiple languages (IDL, Java, MATLAB, and Python). Multiple data providers in the US and Europe have added HAPI access alongside their existing interfaces, and based on this experience, data can be served via HAPI with little or no information loss compared to similar existing web interfaces. Finally, HAPI has been adopted as a recommended COSPAR standard for time series data delivery.

The Space Physics Data Facility (SPDF https://spdf.gsfc.nasa.gov), as the non-solar NASA Heliophysics active final archive, works with current operating missions and the heliophysics community to ingest, preserve and serve a wide range of past and current public science-quality data from the mesosphere into the furthest reach of deep-space exploration. SPDF facilitates scientific analysis of multi-instrument and multi-mission datasets to enhance the science return of the many missions. SPDF develops and maintains the Common Data Format (CDF) and the associated ISTP/SPDF metadata guidelines. SPDF services include CDAWeb, which supports both survey and burst mode data with graphics, listings and data superset/subset functions. All public data held by SPDF are also available for direct file download by HTTPS or FTPS links from the SPDF home page (https://spdf.gsfc.nasa.gov). SPDF is currently receiving and serving from missions including Parker Solar Probe, MMS, Van Allen Probes, THEMIS/ARTEMIS, GOLD, ACE, Cluster, IBEX, Voyager, Geotail, Wind and many others, and >120 Ground-Based investigations. SPDF recently added support for ARASE/ERG and MAVEN as supplementary access at the requests of those missions, and is expecting Solar Orbiter and ICON data. SPDF also operates the multi-mission orbit displays and query services of SSCWeb and the Java-based 4D Orbit Viewer, as well as the Heliophysics Data Portal (HDP) discipline-wide data inventory and access service, and OMNIWeb and COHOWeb for near-Earth and deep-space solar wind plasma, magnetic field, and energetic particle database, respectively.

Interoperability between datasets in Heliophysics and Planetary archives is increasingly important to address complex science questions about space weather and planetary plasma environments. Yet for cross-disciplinary studies, data ingestion is often a tedious, time-consuming process. We have developed the Heliophysics Application Programmer’s Interface (HAPI), a standard specification that captures a lowest common denominator method for accessing time series data. HAPI offers the ability to request data from multiple sources using a single interface, coupled with the ability to get identically formatted data from each source. HAPI has been recognized as a standard by the Committee on Space Research (COSPAR) and has gained adoption at multiple institutions in the US, including Goddard Space Flight Center’s Coordinated Data Analysis Web (GSFC/CDAWeb), the Planetary Data System Planetary Plasma Interactions Node (PDS/PPI), and the Laboratory for Atmospheric and Space Physics (LASP) Interactive Solar Irradiance Data Center (LISIRD). European plasma data centers such as the French Plasma Physics Data Centre (CDPP) and European Space Astronomy Centre (ESAC) are also in the process of adopting HAPI. We present an overview of the HAPI specification and describe how data centers can add HAPI access to their content. We also present how scientists can plot or download HAPI data using Python or using existing analysis tools such as Autoplot (Faden, 2010) and Space Physics Environment Data Analysis Software (SPEDAS) (Angelopoulos, 2019). Faden, J.B., Weigel, R.S., Merka, J. et al. Autoplot: a browser for scientific data on the web. Earth Sci Inform 3, 41–49 (2010). https://doi.org/10.1007/s12145-010-0049-0 Angelopoulos V, Cruce P, Drozdov A, et al. The Space Physics Environment Data Analysis System (SPEDAS). Space Sci Rev. 2019;215(1):9. doi:10.1007/s11214-018-0576-4

The solar wind shows periods of highly Alfvénic activity, where velocity fluctuations and magnetic fluctuations are aligned or antialigned with each other. It is generally agreed that solar wind plasma velocity and magnetic field fluctuations observed by the Parker Solar Probe (PSP) during the first encounter are mostly highly Alfvénic. However, quantitative measures of Alfvénicity are needed to understand how the characterization of these fluctuations compares with standard measures from prior missions in the inner and outer heliosphere, in fast wind and slow wind, and at high and low latitudes. To investigate this issue, we employ several measures to quantify the extent of Alfvénicity—the Alfvén ratio r _A, the normalized cross helicity σ _c , the normalized residual energy σ _r , and the cosine of angle between velocity and magnetic fluctuations . We show that despite the overall impression that the Alfvénicity is large in the solar wind sampled by PSP during the first encounter, during some intervals the cross helicity starts decreasing at very large scales. These length scales (often >1000d _i ) are well inside inertial range, and therefore, the suppression of cross helicity at these scales cannot be attributed to kinetic physics. This drop at large scales could potentially be explained by large scale shears present in the inner heliosphere sampled by PSP. In some cases, despite the cross helicity being constant down to the noise floor, the residual energy decreases with scale in the inertial range. These results suggest that it is important to consider all these measures to quantify Alfvénicity.

Conclusively determining the states of the solar wind will aid in tracing the origins of those states to the Sun, and in the process help to find the wind’s origin and acceleration mechanism(s). Prior studies have characterized the various states of the wind, making lists that are only partially based on objective criteria; different approaches obtain substantially different results. To uncover the unbiased states of the solar wind, we use “k-means clustering”—an unsupervised machine learning method—including constructed multipoint variables. The method allows exploration of different descriptive state variables and numbers of fundamental states (clusters). We show that the clusters reveal structures similar to those found by more ad hoc means, including coronal hole wind, interplanetary coronal mass ejections, “slow wind” (better: noncoronal hole flow), “pseudostreamers,” and stream interaction regions, but with differences that should be useful in refining our previous ideas. These results demonstrate the viability of the approach and warrant further study to understand the origin of remaining discrepancies. Complexity in k-means characterization of the wind may ultimately point to complexity at the source; studies closer to the Sun with Parker Solar Probe will help. We confirm the utility of a set of variables that can serve as a proxy for composition measurements. This proxy permits studies at high time resolution and where composition is not available. This and our recently developed unsupervised multivariate clustering technique are expected to be beneficial in the automated identification of structures and events in a variety of studies.

The Space Physics Data Facility (SPDF https://spdf.gsfc.nasa.gov) and Solar Data Analysis Center (SDAC https://umbra.nascom.nasa.gov/), as the NASA Heliophysics active final archives, will be preserving and distributing the data from Parker Solar Probe. Working in cooperation with current operating missions and the heliophysics community, SPDF ingests, preserves and serves a wide range of past and current public science-quality data from the ionosphere into the furthest reach of deep-space exploration. SPDF has been working with the Parker Solar Probe mission in preparation for archiving and serving its in-situ data starting 2019 Nov 12, and also has arrangements to serve in-situ data from Solar Orbiter when those data become public. SPDF will facilitate scientific analysis of multi-instrument and multi-mission datasets to enhance the science return of Parker Solar Probe mission. SPDF develops and maintains the Common Data Format (CDF) and the associated ISTP/SPDF metadata guidelines. SPDF services include CDAWeb, which supports both survey and burst mode data with graphics, listings and data superset/subset functions. All public data held by SPDF are also available for direct file download by HTTPS or FTPS links from the SPDF home page (https://spdf.gsfc.nasa.gov). SPDF is currently receiving and serving from missions including Helios, MMS, Van Allen Probes, THEMIS/ARTEMIS, GOLD, ACE, Cluster, Geotail, Polar, Wind and many others, and >120 Ground-Based investigations. SPDF recently added support for ARASE/ERG and MAVEN as supplementary access at the requests of those missions. SPDF also operates the multi-mission orbit displays and query services of SSCWeb and the Java-based 4D Orbit Viewer, as well as the Heliophysics Data Portal (HDP) discipline-wide data inventory and access service, and the OMNIweb near-Earth solar wind plasma and magnetic field database.