Tao Shen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Tao Shen has a total of 111 co-authors. Here are the top co-authors by number of shared publications:
- Guodong Long
University of Technology Sydney
21 shared publications - Jing Jiang
University of Technology Sydney
10 shared publications - Xiubo Geng
Microsoft, Beijing, China
9 shared publications - Chongyang Tao
Beihang University
8 shared publications - Daxin Jiang
StepFun, Shanghai, China
7 shared publications - Andrew Yates
Johns Hopkins University, Baltimore, Maryland, USA
5 shared publications
Latest Publications
- Open Access
- 8 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 2883-2886
MedShort: Medicine Shortage Detection on Social Media with Privacy Preservation
- 8 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 2883-2886
Open AccessMedShort: Medicine Shortage Detection on Social Media with Privacy Preservation
- Open Access
- 10 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 934-943
RetriEVAL: Evaluating Text Generation with Contextualized Lexical Match
Pre-trained language models have made significant advancements in text generation tasks. Nevertheless, evaluating the generated text with automatic metrics is still challenging. Compared with supervised metrics, unsupervised metrics which are known for generality and robustness, are frequently employed to assess the quality of generated text efficiently. The representative unsupervised metric BERTScore uses pretrained embedding to calculate the word-to-word similarity across all tokens as evaluation scores, which can introduce potential noise due to the inclusion of tokens that do not contribute significantly to the semantics of the text. Furthermore, its heavy reliance on dense embeddings may lead to lower accuracy when evaluating text outside the common contexts represented in the training data, making it less effective in handling uncommon linguistic patterns Additionally, BERTScore treats all tokens with equal importance and lacks the ability to perform meaningful contextual expansion, which can result in less accurate similarity measurements, particularly when dealing with paraphrased or semantically rich text. To address this problem, we propose an unsupervised automatic evaluation metric inspired by the concept of lexical match in information retrieval. Our method leverages contextualized lexical matching to measure exact matches between identical tokens and dynamically matches different tokens based on their contextualized representations. Experiments on SummEval and Topical-Chat demonstrate our proposed RetriEVAL can correlate better with human judgments than previous unsupervised metrics.
- 10 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 934-943
Open AccessRetriEVAL: Evaluating Text Generation with Contextualized Lexical Match
Pre-trained language models have made significant advancements in text generation tasks. Nevertheless, evaluating the generated text with automatic metrics is still challenging. Compared with supervised metrics, unsupervised metrics which are known for generality and robustness, are frequently employed to assess the quality of generated text efficiently. The representative unsupervised metric BERTScore uses pretrained embedding to calculate the word-to-word similarity across all tokens as evaluation scores, which can introduce potential noise due to the inclusion of tokens that do not contribute significantly to the semantics of the text. Furthermore, its heavy reliance on dense embeddings may lead to lower accuracy when evaluating text outside the common contexts represented in the training data, making it less effective in handling uncommon linguistic patterns Additionally, BERTScore treats all tokens with equal importance and lacks the ability to perform meaningful contextual expansion, which can result in less accurate similarity measurements, particularly when dealing with paraphrased or semantically rich text. To address this problem, we propose an unsupervised automatic evaluation metric inspired by the concept of lexical match in information retrieval. Our method leverages contextualized lexical matching to measure exact matches between identical tokens and dynamically matches different tokens based on their contextualized representations. Experiments on SummEval and Topical-Chat demonstrate our proposed RetriEVAL can correlate better with human judgments than previous unsupervised metrics.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 17772-17781
ThinkQE: Query Expansion via an Evolving Thinking Process
Yibin Lei, Tao Shen, Andrew Yates. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 17772-17781
Open AccessThinkQE: Query Expansion via an Evolving Thinking Process
Yibin Lei, Tao Shen, Andrew Yates. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 18986-19001
Enhancing Lexicon-Based Text Embeddings with Large Language Models
Yibin Lei, Tao Shen, Yu Cao, Andrew Yates. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 18986-19001
Open AccessEnhancing Lexicon-Based Text Embeddings with Large Language Models
Yibin Lei, Tao Shen, Yu Cao, Andrew Yates. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
- Open Access
- 1 August 2024
- Published by International Joint Conferences on Artificial Intelligence
- Vol. 7, 5734-5742
Identifying and Reusing Learnwares Across Different Label Spaces
The learnware paradigm focuses on leveraging numerous established high-performing models to solve machine learning tasks instead of starting from scratch. As the key concept of this paradigm, a learnware consists of a well-trained model of any structure and a specification that characterizes the model's capabilities, allowing it to be identified and reused for future tasks. Given the existence of numerous real-world models trained on diverse label spaces, effectively identifying and combining these models to address tasks involving previously unseen label spaces represents a critical challenge in this paradigm. In this paper, we make the first attempt to identify and reuse effective learnware combinations for tackling learning tasks across different label spaces, extending their applicability beyond the original purposes of individual learnwares. To this end, we introduce a statistical class-wise specification for establishing similarity relations between various label spaces. Leveraging these relations, we model the utility of a learnware combination as a minimum-cost maximum-flow problem, and further develop fine-grained learnware identification and assembly methods. Extensive experiments with thousands of heterogeneous models validate our approach, demonstrating that reusing identified learnware combinations can outperform both training from scratch and fine-tuning a generic pre-trained model.
- 1 August 2024
- Published by International Joint Conferences on Artificial Intelligence
- Vol. 7, 5734-5742
Open AccessIdentifying and Reusing Learnwares Across Different Label Spaces
The learnware paradigm focuses on leveraging numerous established high-performing models to solve machine learning tasks instead of starting from scratch. As the key concept of this paradigm, a learnware consists of a well-trained model of any structure and a specification that characterizes the model's capabilities, allowing it to be identified and reused for future tasks. Given the existence of numerous real-world models trained on diverse label spaces, effectively identifying and combining these models to address tasks involving previously unseen label spaces represents a critical challenge in this paradigm. In this paper, we make the first attempt to identify and reuse effective learnware combinations for tackling learning tasks across different label spaces, extending their applicability beyond the original purposes of individual learnwares. To this end, we introduce a statistical class-wise specification for establishing similarity relations between various label spaces. Leveraging these relations, we model the utility of a learnware combination as a minimum-cost maximum-flow problem, and further develop fine-grained learnware identification and assembly methods. Extensive experiments with thousands of heterogeneous models validate our approach, demonstrating that reusing identified learnware combinations can outperform both training from scratch and fine-tuning a generic pre-trained model.
- Open Access
- 13 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 292-301
MS MARCO Web Search: A Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of downstream tasks and encourages research in various areas, such as generic end-to-end neural indexer models, generic embedding models, and next generation information access system with large language models. MS MARCO Web Search offers a retrieval benchmark with three web retrieval challenge tasks that demands innovations in both machine learning and information retrieval system research domains. As the first dataset that meets large, real and rich data requirements, MS MARCO Web Search paves the way for future advancements in AI and system research. MS MARCO Web Search dataset is available at: https://github.com/microsoft/MS-MARCO-Web-Search.
- 13 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 292-301
Open AccessMS MARCO Web Search: A Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of downstream tasks and encourages research in various areas, such as generic end-to-end neural indexer models, generic embedding models, and next generation information access system with large language models. MS MARCO Web Search offers a retrieval benchmark with three web retrieval challenge tasks that demands innovations in both machine learning and information retrieval system research domains. As the first dataset that meets large, real and rich data requirements, MS MARCO Web Search paves the way for future advancements in AI and system research. MS MARCO Web Search dataset is available at: https://github.com/microsoft/MS-MARCO-Web-Search.
- Open Access
Federated Prompt Learning for Weather Foundation Models on Devices
On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing, holds significance for supporting human activates. Federated Learning is a promising solution for such forecasting by enabling collaborative model training without sharing raw data. However, it faces three main challenges that hinder its reliability: (1) data heterogeneity among devices due to geographic differences; (2) data homogeneity within individual devices and (3) communication overload from sending large model parameters for collaboration. To address these challenges, this paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD), which enables devices to obtain highly customized models while maintaining communication efficiency. Concretely, our Adaptive Prompt Tuning leverages lightweight prompts guide frozen foundation model to generate more precise predictions, also conducts prompt-based multi-level communication to encourage multi-source knowledge fusion and regulate optimization. Additionally, Dynamic Graph Modeling constructs graphs from prompts, prioritizing collaborative training among devices with similar data distributions to against heterogeneity. Extensive experiments demonstrates FedPoD leads the performance among state-of-the-art baselines across various setting in real-world on-device weather forecasting datasets.
Open AccessFederated Prompt Learning for Weather Foundation Models on Devices
On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing, holds significance for supporting human activates. Federated Learning is a promising solution for such forecasting by enabling collaborative model training without sharing raw data. However, it faces three main challenges that hinder its reliability: (1) data heterogeneity among devices due to geographic differences; (2) data homogeneity within individual devices and (3) communication overload from sending large model parameters for collaboration. To address these challenges, this paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD), which enables devices to obtain highly customized models while maintaining communication efficiency. Concretely, our Adaptive Prompt Tuning leverages lightweight prompts guide frozen foundation model to generate more precise predictions, also conducts prompt-based multi-level communication to encourage multi-source knowledge fusion and regulate optimization. Additionally, Dynamic Graph Modeling constructs graphs from prompts, prioritizing collaborative training among devices with similar data distributions to against heterogeneity. Extensive experiments demonstrates FedPoD leads the performance among state-of-the-art baselines across various setting in real-world on-device weather forecasting datasets.
- Open Access
- 24 March 2024
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (17) , 19732-19740
Fine-Grained Distillation for Long Document Retrieval
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the \textit{scope hypothesis} that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.
- 24 March 2024
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (17) , 19732-19740
Open AccessFine-Grained Distillation for Long Document Retrieval
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the \textit{scope hypothesis} that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.
- Open Access
Corpus-Steered Query Expansion with Large Language Models
Recent studies demonstrate that query expansions generated by large language models (LLMs) can considerably enhance information retrieval systems by generating hypothetical documents that answer the queries as expansions. However, challenges arise from misalignments between the expansions and the retrieval corpus, resulting in issues like hallucinations and outdated information due to the limited intrinsic knowledge of LLMs. Inspired by Pseudo Relevance Feedback (PRF), we introduce Corpus-Steered Query Expansion (CSQE) to promote the incorporation of knowledge embedded within the corpus. CSQE utilizes the relevance assessing capability of LLMs to systematically identify pivotal sentences in the initially-retrieved documents. These corpus-originated texts are subsequently used to expand the query together with LLM-knowledge empowered expansions, improving the relevance prediction between the query and the target documents. Extensive experiments reveal that CSQE exhibits strong performance without necessitating any training, especially with queries for which LLMs lack knowledge.
Open AccessCorpus-Steered Query Expansion with Large Language Models
Recent studies demonstrate that query expansions generated by large language models (LLMs) can considerably enhance information retrieval systems by generating hypothetical documents that answer the queries as expansions. However, challenges arise from misalignments between the expansions and the retrieval corpus, resulting in issues like hallucinations and outdated information due to the limited intrinsic knowledge of LLMs. Inspired by Pseudo Relevance Feedback (PRF), we introduce Corpus-Steered Query Expansion (CSQE) to promote the incorporation of knowledge embedded within the corpus. CSQE utilizes the relevance assessing capability of LLMs to systematically identify pivotal sentences in the initially-retrieved documents. These corpus-originated texts are subsequently used to expand the query together with LLM-knowledge empowered expansions, improving the relevance prediction between the query and the target documents. Extensive experiments reveal that CSQE exhibits strong performance without necessitating any training, especially with queries for which LLMs lack knowledge.
- Open Access
CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification
Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, inspired by the human-decision process that the most ambiguous classes would be mulled over for each instance, we propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification. We conduct experiments on many-class benchmark datasets in both the fully supervised setting and the few-shot setting, which indicates that our model outperforms former baselines.
Open AccessCCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification
Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, inspired by the human-decision process that the most ambiguous classes would be mulled over for each instance, we propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification. We conduct experiments on many-class benchmark datasets in both the fully supervised setting and the few-shot setting, which indicates that our model outperforms former baselines.