Regression for Astronomical Data with Realistic Distributions, Errors, and Nonlinearity
Open Access
- 23 June 2025
- journal article
- Published by American Astronomical Society in The Astronomical Journal
- Vol. 170 (1) , 45
Abstract
We have developed a new regression technique, the maximum likelihood (ML)–based method and its variant, the Kolmogorov–Smirnov (KS) test–based method, designed to obtain unbiased regression results from typical astronomical data. A normalizing flow model is employed to automatically estimate the unobservable intrinsic distribution of the independent variable and the unobservable correlation between uncertainty level and intrinsic value of both independent and dependent variables from the observed data points in a variational-inference-based empirical Bayes approach. By incorporating these estimated distributions, our method comprehensively accounts for the uncertainties associated with both independent and dependent variables. Our test on both mock data and real astronomical data from PHANGS-ALMA and PHANGS-JWST demonstrates that, given a sufficiently large sample size (>1000), both the ML-based method and the KS-test-based method significantly outperform the existing widely used methods, particularly in cases of low signal-to-noise ratios. The KS-test-based method exhibits remarkable robustness against deviations from underlying assumptions, complex intrinsic distributions, varying correlations between uncertainty levels and intrinsic values, inaccuracies in uncertainty estimations, outliers, and saturation effects. For sample sizes between 300 and 1000, the ML-based method yields the best performance. In the low-data regime (<300), the ML-based method maintains comparable performance to other state-of-the-art methods. A GPU-compatible Python implementation of our methods, nicknamed “raddest,” will be made publicly available upon acceptance of this paper.Keywords
All Related Versions
Funding Information
- MOST ∣ National Natural Science Foundation of China (12433003)
- MOST ∣ National Key Research and Development Program of China (2022YFA1602902)
This publication has 21 references indexed in Scilit:
- The ALMaQUEST Survey – V. The non-universality of kpc-scale star formation relations and the factors that drive themMonthly Notices of the Royal Astronomical Society, 2020
- LEO-Py: Estimating likelihoods for correlated, censored, and uncertain data with given marginal distributionsAstronomy and Computing, 2019
- Hyper-Fit: Fitting Linear Models to Multidimensional Data with Multivariate Gaussian UncertaintiesPublications of the Astronomical Society of Australia, 2015
- The ATLAS3D project – XV. Benchmark for early-type galaxies scaling relations from 260 dynamical models: mass-to-light ratio, dark matter, Fundamental Plane and Mass PlaneMonthly Notices of the Royal Astronomical Society, 2013
- Extreme deconvolution: Inferring complete distribution functions from noisy, heterogeneous and incomplete observationsThe Annals of Applied Statistics, 2011
- Some Aspects of Measurement Error in Linear Regression of Astronomical DataThe Astrophysical Journal, 2007
- Linear Regression for Astronomical Data with Measurement Errors and Intrinsic ScatterThe Astrophysical Journal, 1996
- Linear regression in astronomy.The Astrophysical Journal, 1990
- A multidimensional version of the Kolmogorov–Smirnov testMonthly Notices of the Royal Astronomical Society, 1987
- Two-dimensional goodness-of-fit testing in astronomyMonthly Notices of the Royal Astronomical Society, 1983