Xiaosheng Zhao

Cross-survey generalization is a critical challenge in stellar spectral analysis, particularly in cases such as transferring from low- to moderate-resolution surveys. We investigate this problem using pretrained models, focusing on simple neural networks such as multilayer perceptrons (MLPs), with a case study transferring from LAMOST low-resolution spectra (LRS) to DESI medium-resolution spectra. Specifically, we pretrain MLPs on either LRS or their embeddings and fine-tune them for application to DESI stellar spectra. We compare MLPs trained directly on spectra with those trained on embeddings derived from transformer-based models (self-supervised foundation models pretrained for multiple downstream tasks). We also evaluate different fine-tuning strategies, including residual-head fine-tuning, low-rank adaptation, and full fine-tuning. We find that MLPs pretrained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results. For iron abundance, embeddings from a transformer-based model yield advantages in the metal-rich ([Fe/H] > −1.0) regime, but underperform in the metal-poor regime compared to MLPs trained directly on LRS. We also show that the optimal fine-tuning strategy depends on the specific stellar parameter under consideration. These results highlight that simple pretrained MLPs can provide competitive cross-survey generalization, while the role of spectral foundation models for cross-survey stellar parameter estimation requires further exploration.

The advent of increasingly advanced surveys and cosmic tracers has motivated the development of new inference techniques and novel approaches to extracting information from cosmic fields. A central challenge in this endeavor is to quantify the information content carried by these summary statistics in cosmic fields. In particular, how should we assess which statistics are more informative than others and assess the exact degree of complementarity of the information from each statistic? Here, we introduce mutual information (MI), which provides, from an information-theoretic perspective, a natural framework for assessing the sufficiency and complementarity of summary statistics in cosmological data. We demonstrate how MI can be applied to typical inference tasks to make information-theoretic evaluations, using two representative examples: the cosmic microwave background map, from which the power spectrum (PS) extracts almost all information as is expected for a Gaussian random field, and the 21 cm brightness temperature map, from which the scattering transform extracts the most non-Gaussian information but which is complementary to the PS and bispectrum. Our results suggest that MI offers a robust theoretical foundation for evaluating and improving summaries, thereby enabling a deeper understanding of cosmic fields from an information-theoretic perspective.

Cross-survey generalization is a critical challenge in stellar spectral analysis, particularly in cases such as transferring from low- to moderate-resolution surveys. We investigate this problem using pre-trained models, focusing on simple neural networks such as multilayer perceptrons (MLPs), with a case study transferring from LAMOST low-resolution spectra (LRS) to DESI medium-resolution spectra (MRS). Specifically, we pre-train MLPs on either LRS or their embeddings and fine-tune them for application to DESI stellar spectra. We compare MLPs trained directly on spectra with those trained on embeddings derived from transformer-based models (self-supervised foundation models pre-trained for multiple downstream tasks). We also evaluate different fine-tuning strategies, including residual-head adapters, LoRA, and full fine-tuning. We find that MLPs pre-trained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results. For iron abundance, embeddings from a transformer-based model yield advantages in the metal-rich ([Fe/H] > -1.0) regime, but underperform in the metal-poor regime compared to MLPs trained directly on LRS. We also show that the optimal fine-tuning strategy depends on the specific stellar parameter under consideration. These results highlight that simple pre-trained MLPs can provide competitive cross-survey generalization, while the role of spectral foundation models for cross-survey stellar parameter estimation requires further exploration.

In recent years, large language models (LLMs) have transformed natural language understanding through vast data sets and large-scale parameterization. Inspired by this success, we present SpecCLIP, a foundation model framework that extends LLM-inspired methodologies to stellar spectral analysis. Stellar spectra, akin to structured language, encode rich physical and chemical information about stars. By training foundation models on large-scale spectral data sets, our goal is to learn robust and informative embeddings that support diverse downstream applications. As a proof of concept, SpecCLIP involves pretraining on two spectral types—LAMOST low-resolution and Gaia XP—followed by contrastive alignment using the Contrastive Language–Image Pretraining (CLIP) framework, adapted to associate spectra from different instruments. This alignment is complemented by auxiliary decoders that preserve spectrum-specific information and enable translation (prediction) between spectral types, the former being achieved by maximizing mutual information between embeddings and input spectra. The result is a cross-spectrum framework that enables intrinsic calibration and flexible applications across instruments. We demonstrate that fine-tuning these models on moderate-sized labeled data sets improves adaptability to tasks such as stellar-parameter estimation and chemical-abundance determination. SpecCLIP also enhances the accuracy and precision of parameter estimates benchmarked against external survey data. In addition, its similarity search and cross-spectrum prediction capabilities offer potential for anomaly detection. Our results suggest that contrastively trained foundation models enriched with spectrum-aware decoders can advance precision stellar spectroscopy. Our code SpecCLIP is publicly available on GitHub ✎.

The advent of increasingly advanced surveys and cosmic tracers has motivated the development of new inference techniques and novel approaches to extracting information from cosmic fields. A central challenge in this endeavor is to quantify the information content carried by these summary statistics in cosmic fields. In particular, how should we assess which statistics are more informative than others and assess the exact degree of complementarity of the information from each statistic? Here, we introduce mutual information (MI) that provides, from an information-theoretic perspective, a natural framework for assessing the sufficiency and complementarity of summary statistics in cosmological data. We demonstrate how MI can be applied to typical inference tasks to make information-theoretic evaluations, using two representative examples: the cosmic microwave background map, from which the power spectrum extracts almost all information as is expected for a Gaussian random field, and the 21~cm brightness temperature map, from which the scattering transform extracts the most non-Gaussian information but is complementary to power spectrum and bispectrum. Our results suggest that MI offers a robust theoretical foundation for evaluating and improving summaries, thereby enabling a deeper understanding of cosmic fields from an information-theoretic perspective.

We present the information-ordered bottleneck (IOB), a neural layer designed to adaptively compress data into latent variables ordered by likelihood maximization. Without retraining, IOB nodes can be truncated at any bottleneck width, capturing the most crucial information in the first latent variables. Unifying several prior approaches, we demonstrate that IOB models achieve efficient compression of essential information for a given encoding architecture, while also assigning a semantically meaningful ordering to latent representations. IOBs demonstrate a remarkable ability to compress embeddings of high-dimensional image and text data, leveraging the performance of SOTA architectures such as CNNs, transformers, and diffusion models. Moreover, we introduce a novel theory for estimating global intrinsic dimensionality with IOBs and show that they recover SOTA dimensionality estimates for complex synthetic data. Furthermore, we showcase the utility of these models for exploratory analysis through applications on heterogeneous datasets, enabling computer-aided discovery of dataset complexity.

We explore likelihood-free (aka simulation-based) Bayesian model selection to quantify model comparison analyses of reionisation scenarios. We iteratively train the 3D Convolutional Neural Network (CNN) on four toy EoR models based on 21cmFAST simulations with contrasting morphology to obtain summaries of the 21 cm lightcone. Within the pyDelfi framework, we replaced the Emcee sampler with MultiNest to integrate learnt posteriors and produce the Bayesian Evidence. We comfortably distinguish the model used to produce the mock data set in all cases. However, we struggle to produce accurate posterior distributions for outside-in reionisation models. After a variety of cross-checks and alternate analyses we discuss the flexibility of summarising models that differ from precisely the intended network training conditions as this should be more widely scrutinised before CNN can reliably analyse observed data.

The information regarding how the intergalactic medium is reionized by astrophysical sources is contained in the tomographic three-dimensional 21 cm images from the epoch of reionization. In Zhao et al. (“Paper I”), we demonstrated for the first time that density estimation likelihood-free inference (DELFI) can be applied efficiently to perform a Bayesian inference of the reionization parameters from the 21 cm images. Nevertheless, the 3D image data needs to be compressed into informative summaries as the input of DELFI by, e.g., a trained 3D convolutional neural network (CNN) as in Paper I (DELFI-3D CNN). Here in this paper, we introduce an alternative data compressor, the solid harmonic wavelet scattering transform (WST), which has a similar, yet fixed (i.e., no training), architecture to CNN, but we show that this approach (i.e., solid harmonic WST with DELFI) outperforms earlier analyses based on 3D 21 cm images using DELFI-3D CNN in terms of credible regions of parameters. Realistic effects, including thermal noise and residual foreground after removal, are also applied to the mock observations from the Square Kilometre Array. We show that under the same inference strategy using DELFI, the 21 cm image analysis with solid harmonic WST outperforms the 21 cm power spectrum analysis. This research serves as a proof of concept, demonstrating the potential to harness the strengths of WST and simulation-based inference to derive insights from future 21 cm light-cone image data.

Generative adversarial networks (GANs) are frequently utilized in astronomy to construct an emulator of numerical simulations. Nevertheless, training GANs can prove to be a precarious task, as they are prone to instability and often lead to mode collapse problems. Conversely, the diffusion model also has the ability to generate high-quality data without adversarial training. It has shown superiority over GANs with regard to several natural image datasets. In this study, we undertake a quantitative comparison between the denoising diffusion probabilistic model (DDPM) and StyleGAN2 (one of the most robust types of GANs) via a set of robust summary statistics from scattering transform. In particular, we utilize both models to generate the images of 21 cm brightness temperature mapping, as a case study, conditionally based on astrophysical parameters that govern the process of cosmic reionization. Using our new Fréchet Scattering Distance (FSD) as the evaluation metric to quantitatively compare the sample distribution between generative models and simulations, we demonstrate that DDPM outperforms StyleGAN2 on varied sizes of training sets. Through Fisher forecasts, we demonstrate that on our datasets, StyleGAN2 exhibits mode collapses in varied ways, while DDPM yields a more robust generation. We also explore the role of classifier-free guidance in DDPM and show the preference for a non-zero guidance scale only when the training data is limited. Our findings indicate that the diffusion model presents a promising alternative to GANs in the generation of accurate images. These images can subsequently provide reliable parameter constraints, particularly in the realm of astrophysics.

Generative adversarial networks (GANs) are frequently utilized in astronomy to construct an emulator of numerical simulations. Nevertheless, training GANs can prove to be a precarious task, as they are prone to instability and often lead to mode collapse problems. Conversely, the diffusion model also has the ability to generate high-quality data without adversarial training. It has shown superiority over GANs with regard to several natural image data sets. In this study, we undertake a quantitative comparison between the denoising diffusion probabilistic model (DDPM) and StyleGAN2 (one of the most robust types of GANs) via a set of robust summary statistics from scattering transform. In particular, we utilize both models to generate the images of 21 cm brightness temperature mapping, as a case study, conditionally based on astrophysical parameters that govern the process of cosmic reionization. Using our new Fréchet Scattering Distance (FSD) as the evaluation metric to quantitatively compare the sample distribution between generative models and simulations, we demonstrate that DDPM outperforms StyleGAN2 on varied sizes of training sets. Through Fisher forecasts, we demonstrate that on our data sets, StyleGAN2 exhibits mode collapses in varied ways, while DDPM yields a more robust generation. We also explore the role of classifier-free guidance in DDPM and show the preference for a non-zero guidance scale only when the training data is limited. Our findings indicate that the diffusion model presents a promising alternative to GANs in the generation of accurate images. These images can subsequently provide reliable parameter constraints, particularly in the realm of astrophysics.