Tong Xiao

Ordinary differential equations (ODEs) provide a mathematical foundation for understanding the continuous evolution of complex systems. While deep learning has traditionally relied on discrete layer-wise computations, many state-of-the-art models in vision and language can be fundamentally interpreted as discretized approximations of continuous dynamical processes. In this paper, we present a unified theoretical framework that aligns diverse neural architectures and generative processes through the formalism of ODEs. We demonstrate that this continuoustime perspective offers a principled approach to model design across three progressive dimensions. First, we show how classic residual networks and Transformers can be formulated as specific discretization schemes of underlying ODEs, motivating the design of advanced architectures. Second, we analyze neural ODEs and flow-based models, and discuss how fully continuous-time formulations enable more flexible density estimation and parameter efficiency. Third, we treat diffusion models as a connection between discrete and continuous frameworks. We show that modern generative models in vision and language can be interpreted from this unified perspective of differential equations. More broadly, ODEs can be used as a general paradigm for posing and analyzing deep learning problems in vision and language. This perspective can, in turn, drive new architecture and algorithm designs by leveraging the powerful mathematical toolkit of continuous dynamical systems.

Preference optimization methods like DPO have achieved remarkable performance in LLM alignment. However, the evaluation for these methods relies on a single response and overlooks other potential outputs, which could also be generated in real-world applications within this hypothetical space. To address this issue, this paper presents a \textbf{H}ypothesis-based Pr\textbf{E}ference-aware \textbf{A}na\textbf{L}ysis Framework (HEAL), a novel evaluation paradigm that formulates preference alignment as a re-ranking process within hypothesis spaces. The framework incorporates two complementary metrics: ranking accuracy for evaluating ordinal consistency and preference strength correlation for assessing continuous alignment. To facilitate this framework, we develop UniHypoBench, a unified hypothesis benchmark constructed from diverse instruction-response pairs. Through extensive experiments based on HEAL, with a particular focus on the intrinsic mechanisms of preference learning, we demonstrate that current preference learning methods can effectively capture preferences provided by proxy models while simultaneously suppressing negative samples. These findings contribute to preference learning research through two significant avenues. Theoretically, we introduce hypothesis space analysis as an innovative paradigm for understanding preference alignment. Practically, HEAL offers researchers robust diagnostic tools for refining preference optimization methods, while our empirical results identify promising directions for developing more advanced alignment algorithms capable of comprehensive preference capture.

Recent advances in speech large language models (SLMs) have improved speech recognition and translation in general domains, but accurately generating domain-specific terms or neologisms remains challenging. To address this, we propose Attention2Probability: attention-driven terminology probability estimation for robust speech-to-text system, which is lightweight, flexible, and accurate. Attention2Probability converts cross-attention weights between speech and terminology into presence probabilities, and it further employs curriculum learning to enhance retrieval accuracy. Furthermore, to tackle the lack of data for speech-to-text tasks with terminology intervention, we create and release a new speech dataset with terminology to support future research in this area. Experimental results show that Attention2Probability significantly outperforms the VectorDB method on our test set. Specifically, its maximum recall rates reach 92.57% for Chinese and 86.83% for English. This high recall is achieved with a latency of only 8.71ms per query. Intervening in SLMs' recognition and translation tasks using Attention2Probability-retrieved terms improves terminology accuracy by 6-17%, while revealing that the current utilization of terminology by SLMs has limitations.

Model merging has emerged as a compelling data-free paradigm for multi-task learning, enabling the fusion of multiple fine-tuned models into a single, powerful entity. A key technique in merging methods is sparsification, which prunes redundant parameters from task vectors to mitigate interference. However, prevailing approaches employ a ``one-size-fits-all'' strategy, applying a uniform sparsity ratio that overlooks the inherent structural and statistical heterogeneity of model parameters. This often leads to a suboptimal trade-off, where critical parameters are inadvertently pruned while less useful ones are retained. To address this limitation, we introduce \textbf{TADrop} (\textbf{T}ensor-wise \textbf{A}daptive \textbf{Drop}), an adaptive sparsification strategy that respects this heterogeneity. Instead of a global ratio, TADrop assigns a tailored sparsity level to each parameter tensor based on its distributional properties. The core intuition is that tensors with denser, more redundant distributions can be pruned aggressively, while sparser, more critical ones are preserved. As a simple and plug-and-play module, we validate TADrop by integrating it with foundational, classic, and SOTA merging methods. Extensive experiments across diverse tasks (vision, language, and multimodal) and models (ViT, BEiT) demonstrate that TADrop consistently and significantly boosts their performance. For instance, when enhancing a leading merging method, it achieves an average performance gain of 2.0\% across 8 ViT-B/32 tasks. TADrop provides a more effective way to mitigate parameter interference by tailoring sparsification to the model's structure, offering a new baseline for high-performance model merging.

Automatic evaluation of sequence generation, which has traditionally relied on metrics such as BLEU and ROUGE, often struggles to capture the semantic accuracy of generated text due to an overemphasis on n-gram overlap. A promising solution to this issue is the development of model-based metrics, such as BLEURT and COMET. However, these approaches are typically limited by the scarcity of labeled evaluation data, which is essential for training evaluation models. In this work, we address this challenge by proposing the Customized Sequence Evaluation Metric (CSEM), a three-stage model training method that leverages large language models to generate labeled data for metric development, eliminating the need for human-labeled data. Furthermore, we extend the capabilities of CSEM to support a range of evaluation types, including single-aspect, multi-aspect, reference-free, and reference-based evaluations. This flexibility allows for the customization of metrics to fit various real-world scenarios. Experimental results on the SummEval benchmark demonstrate that CSEM can effectively train an evaluation model without human-labeled data. Additional experiments in reinforcement learning and reranking show that metrics developed through CSEM outperform traditional evaluation metrics, leading to significant improvements in sequence quality, as assessed by both commonly used metrics and ChatGPT.

The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using $n$-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.

The Transformer model has achieved notable success in various sequence generation tasks such as machine translation and generative text summarization. However, its performance remains suboptimal in low-resource scenarios. This limitation stems from insufficient learning of model parameters and the reliance on a single-scale, global modeling mechanism. Inspired by multi-scale approaches in computer vision and the dot-product mechanism in Transformer attention, we propose a simple yet parameter-efficient method to enhance the model’s learning capacity without increasing its parameter count. The method involves constructing multi-scale features with diverse attributes from the model’s hidden dimension through matrix decomposition-based sampling, followed by the fusion of attention weights generated from these multi-scale features. This fusion results in a more robust data representation, allowing the model to learn from the data more effectively. Experimental results show that our method outperforms baseline approaches by an average of 0.45 BLEU points across six machine translation tasks and achieves a 0.55 ROUGE point improvement on the CNN-DailyMail summarization test set.