David Shih

Next token prediction is an attractive pre-training task for jet foundation models, in that it is simulation free and enables excellent generative capabilities that can transfer across datasets. Here we study multiple improvements to next token prediction, building on the initial work of OmniJet-α. Instead of tokenizing particles and subsequently only using the token-ID as the model input for both the generative and the classification task, we adopt a hybrid setup, which allows us to use continuous feature vectors as model input while only using token-IDs in the next token prediction target. Secondly, we explore a combined pre-training strategy that combines masked particle modeling and generative learning objectives. Taken together, these changes greatly improve the performance in downstream classification tasks without any loss in generative performance.

An increasing number of discoveries of isolated and quenched dwarf galaxies are challenging the idea that the present-day local environment of low-mass systems is the main determinant of their quenching. We present new Hubble Space Telescope (HST) data of one such system, the dwarf galaxy Canes Venatici C (CVnC). CVnC is a low-mass ( 3 . 4 − 2.6 + 4.2 × 1 0 6 M ⊙ ) galaxy with a Tip of the Red Giant Branch distance of 8.4 3 − 0.32 + 0.47 Mpc determined from the resolved stars in the HST imaging, which we also use to derive CVnC’s structural parameters. CVnC’s distance places CVnC in the Local Volume and in an isolated environment with the most tidally influential L _⋆ galaxy > 5 R _vir away. Additional constraints from the HST color–magnitude diagram, archival far-ultraviolet, and neutral hydrogen (H i ) data show that CVnC is quenched, with no evidence of star formation in the last 100 Myr and no detectable gas (M _{H i} < 1.5 × 10 ⁶ M _⊙ ). Circumstantial evidence suggests that CVnC may have quenched via past interactions with the L _⋆ galaxy NGC 4631 ( L _K = 10 ^10.4 L _⊙ ), and was possibly sent on an extreme backsplash orbit by the tidal dissolution of a subhalo group. However, other quenching mechanisms—such as stripping via the cosmic web—cannot be ruled out. CVnC adds to the growing number of quenched dwarf galaxies in underdense environments, a population that will be critical to defining the mass and environment regimes in which different quenching mechanisms operate.

We propose searching for physics beyond the Standard Model in the low-transverse-momentum tracks accompanying hard-scatter events at the LHC. TeV-scale resonances connected to a dark QCD sector could be enhanced by selecting events with anomalies in the track distributions. As a benchmark, a quirk model with microscopic string lengths is developed, including a setup for event simulation. For this model, strategies are presented to enhance the sensitivity compared to inclusive resonance searches: a simple cut-based selection, a supervised search, and a model-agnostic weakly supervised anomaly search with the CATHODE method. Expected discovery potentials and exclusion limits are shown for 140 fb − 1 of 13 TeV proton-proton collisions at the LHC.

Machine learning-based anomaly detection methods are able to search high-dimensional spaces for hints of new physics with much less theory bias than traditional searches. However, by searching in many directions all at once, the statistical power of these search strategies is diluted by a variant of the look elsewhere effect. We examine this challenge in detail, focusing on weakly supervised methods. We find that training and testing on the same data results in badly miscalibrated $p$-values due to the anomaly detector searching everywhere in the data and overfitting on statistical fluctuations. However, if these $p$-values can be calibrated, they may offer the best sensitivity to anomalies, since this approach uses all of the data. Conversely, training on half of the data and testing on the other half results in perfectly calibrated $p$-values, but at the cost of reduced sensitivity to anomalies. Similarly, regularization methods such as early stopping can help with $p$-value calibration but also possibly at the expense of sensitivity. Finally, we find that k-folding strikes an effective balance between calibration and sensitivity. Our findings are supported by numerical studies with Gaussian random variables as well as from collider physics using the LHC Olympics benchmark anomaly detection dataset.

Next token prediction is an attractive pre-training task for jet foundation models, in that it is simulation free and enables excellent generative capabilities that can transfer across datasets. Here we study multiple improvements to next token prediction, building on the initial work of OmniJet-$α$. Instead of tokenizing particles and subsequently only using the token-ID as the model input for both the generative and the classification task, we adopt a hybrid setup, which allows us to use continuous feature vectors as model input while only using token-IDs in the next token prediction target. Secondly, we explore a combined pre-training strategy that combines masked particle modeling and generative learning objectives. Taken together, these changes greatly improve the performance in downstream classification tasks without any loss in generative performance.

Beyond the practical goal of improving search and measurement sensitivity through better jet tagging algorithms, there is a deeper question: what are their upper performance limits? Generative surrogate models with learned likelihood functions offer a new approach to this problem, provided the surrogate correctly captures the underlying data distribution. In this work, we introduce the SUrrogate ReFerence (SURF) method, a new approach to validating generative models. This framework enables exact Neyman-Pearson tests by training the target model on samples from another tractable surrogate, which is itself trained on real data. We argue that the EPiC-FM generative model is a valid surrogate reference for JetClass jets and apply SURF to show that modern jet taggers may already be operating close to the true statistical limit. By contrast, we find that autoregressive GPT models unphysically exaggerate top vs. QCD separation power encoded in the surrogate reference, implying that they are giving a misleading picture of the fundamental limit.

We present the results of the ‘Fast Calorimeter Simulation Challenge 2022’—the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including variational autoencoders (VAEs), generative adversarial networks (GANs), normalizing flows, diffusion models, and models based on conditional flow matching. We compare all submissions in terms of quality of generated calorimeter showers, as well as shower generation time and model size. To assess the quality we use a broad range of different metrics including differences in one-dimensional histograms of observables, KPD/FPD scores, AUCs of binary classifiers, and the log-posterior of a multiclass classifier. The results of the CaloChallenge provide the most complete and comprehensive survey of cutting-edge approaches to calorimeter fast simulation to date. In addition, our work provides a uniquely detailed perspective on the important problem of how to evaluate generative models. As such, the results presented here should be applicable for other domains that use generative AI and require fast and faithful generation of samples in a large phase space. Report Numbers : HEPHY-ML-24-05, FERMILAB-PUB-24-0728-CMS, TTK-24-43.

A key step in any resonant anomaly detection search is accurate modeling of the background distribution in each signal region. Data-driven methods like CATHODE accomplish this by training separate generative models on the complement of each signal region, and interpolating them into their corresponding signal regions. Having to retrain the generative model on essentially the entire dataset for each signal region is a major computational cost in a typical sliding window search with many signal regions. Here, we present SIGMA, a new, fully data-driven, computationally efficient method for estimating background distributions. The idea is to train a single generative model on all of the data and interpolate its parameters in sideband regions in order to obtain a model for the background in the signal region. The SIGMA method significantly reduces the computational cost compared to previous approaches, while retaining a similar high quality of background modeling and sensitivity to anomalous signals.

Foundation models are deep learning models pre-trained on large amounts of data which are capable of generalizing to multiple datasets and/or downstream tasks. This work demonstrates how data collected by the CMS experiment at the Large Hadron Collider can be useful in pre-training foundation models for HEP. Specifically, we introduce the AspenOpenJets (AOJs) dataset, consisting of approximately 178 M high p T jets derived from CMS 2016 Open Data. We show how pre-training the OmniJet-α foundation model on AOJs improves performance on generative tasks with significant domain shift: generating boosted top and QCD jets from the simulated JetClass dataset. In addition to demonstrating the power of pre-training of a jet-based foundation model on actual proton–proton collision data, we provide the ML-ready derived AOJs dataset for further public use.

Statistical inference in physics is often based on samples from a generator (sometimes referred to as a ``forward model") that emulate experimental data and depend on parameters of the underlying theory. Modern machine learning has supercharged this workflow to enable high-dimensional and unbinned analyses to utilize much more information than ever before. We propose a general framework for describing the integration of machine learning with generators called Generator Based Inference (GBI). A well-studied special case of this setup is Simulation Based Inference (SBI) where the generator is a physics-based simulator. In this work, we examine other methods within the GBI toolkit that use data-driven methods to build the generator. In particular, we focus on resonant anomaly detection, where the generator describing the background is learned from sidebands. We show how to perform machine learning-based parameter estimation in this context with data-derived generators. This transforms the statistical outputs of anomaly detection to be directly interpretable and the performance on the LHCO community benchmark dataset establishes a new state-of-the-art for anomaly detection sensitivity.