Yangdong Deng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Interests
Yangdong Deng has a total of 69 co-authors. Here are the top co-authors by number of shared publications:
- Wojciech P. Maly
Electrical and Computer Engineering Carnegie Mellon University
8 shared publications Ming Gu
School of Software, Tsinghua University, China
5 shared publications- Zonghui Li
Beijing Jiaotong University
5 shared publications - Hai Wan
Tsinghua University, Beijing, China
4 shared publications - Jian Sun
Megvii Inc
3 shared publications - Jin Huang
Tsinghua University
3 shared publications
Latest Publications
- Open Access
A Multi-Level Framework for Accelerating Training Transformer Models
The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other domains. Training such models, however, poses an unprecedented demand for computing power, which incurs exponentially increasing energy cost and carbon dioxide emissions. It is thus critical to develop efficient training solutions to reduce the training costs. Motivated by a set of key observations of inter- and intra-layer similarities among feature maps and attentions that can be identified from typical training processes, we propose a multi-level framework for training acceleration. Specifically, the framework is based on three basic operators, Coalescing, De-coalescing and Interpolation, which can be orchestrated to build a multi-level training framework. The framework consists of a V-cycle training process, which progressively down- and up-scales the model size and projects the parameters between adjacent levels of models via coalescing and de-coalescing. The key idea is that a smaller model that can be trained for fast convergence and the trained parameters provides high-qualities intermediate solutions for the next level larger network. The interpolation operator is designed to break the symmetry of neurons incurred by de-coalescing for better convergence performance. Our experiments on transformer-based language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model while preserving the performance.
Open AccessA Multi-Level Framework for Accelerating Training Transformer Models
The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other domains. Training such models, however, poses an unprecedented demand for computing power, which incurs exponentially increasing energy cost and carbon dioxide emissions. It is thus critical to develop efficient training solutions to reduce the training costs. Motivated by a set of key observations of inter- and intra-layer similarities among feature maps and attentions that can be identified from typical training processes, we propose a multi-level framework for training acceleration. Specifically, the framework is based on three basic operators, Coalescing, De-coalescing and Interpolation, which can be orchestrated to build a multi-level training framework. The framework consists of a V-cycle training process, which progressively down- and up-scales the model size and projects the parameters between adjacent levels of models via coalescing and de-coalescing. The key idea is that a smaller model that can be trained for fast convergence and the trained parameters provides high-qualities intermediate solutions for the next level larger network. The interpolation operator is designed to break the symmetry of neurons incurred by de-coalescing for better convergence performance. Our experiments on transformer-based language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model while preserving the performance.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 7293-7307
CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers
Longwei Zou, Qingyang Wang, Han Zhao, Jiangangkong Jiangangkong, Yi Yang, Yangdong Deng. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 7293-7307
Open AccessCQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers
Longwei Zou, Qingyang Wang, Han Zhao, Jiangangkong Jiangangkong, Yi Yang, Yangdong Deng. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- Open Access
- 20 December 2023
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 11, 1608-1623
Removing Backdoors in Pre-trained Models by Regularized Continual Pre-training
Recent research has revealed that pre-trained models (PTMs) are vulnerable to backdoor attacks before the fine-tuning stage. The attackers can implant transferable task-agnostic backdoors in PTMs, and control model outputs on any downstream task, which poses severe security threats to all downstream applications. Existing backdoor-removal defenses focus on task-specific classification models and they are not suitable for defending PTMs against task-agnostic backdoor attacks. To this end, we propose the first task-agnostic backdoor removal method for PTMs. Based on the selective activation phenomenon in backdoored PTMs, we design a simple and effective backdoor eraser, which continually pre-trains the backdoored PTMs with a regularization term in an end-to-end approach. The regularization term removes backdoor functionalities from PTMs while the continual pre-training maintains the normal functionalities of PTMs. We conduct extensive experiments on pre-trained models across different modalities and architectures. The experimental results show that our method can effectively remove backdoors inside PTMs and preserve benign functionalities of PTMs with a few downstream-task-irrelevant auxiliary data, e.g., unlabeled plain texts. The average attack success rate on three downstream datasets is reduced from 99.88% to 8.10% after our defense on the backdoored BERT. The codes are publicly available at https://github.com/thunlp/RECIPE.
- 20 December 2023
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 11, 1608-1623
Open AccessRemoving Backdoors in Pre-trained Models by Regularized Continual Pre-training
Recent research has revealed that pre-trained models (PTMs) are vulnerable to backdoor attacks before the fine-tuning stage. The attackers can implant transferable task-agnostic backdoors in PTMs, and control model outputs on any downstream task, which poses severe security threats to all downstream applications. Existing backdoor-removal defenses focus on task-specific classification models and they are not suitable for defending PTMs against task-agnostic backdoor attacks. To this end, we propose the first task-agnostic backdoor removal method for PTMs. Based on the selective activation phenomenon in backdoored PTMs, we design a simple and effective backdoor eraser, which continually pre-trains the backdoored PTMs with a regularization term in an end-to-end approach. The regularization term removes backdoor functionalities from PTMs while the continual pre-training maintains the normal functionalities of PTMs. We conduct extensive experiments on pre-trained models across different modalities and architectures. The experimental results show that our method can effectively remove backdoors inside PTMs and preserve benign functionalities of PTMs with a few downstream-task-irrelevant auxiliary data, e.g., unlabeled plain texts. The average attack success rate on three downstream datasets is reduced from 99.88% to 8.10% after our defense on the backdoored BERT. The codes are publicly available at https://github.com/thunlp/RECIPE.
- Open Access
TCGAN: Convolutional Generative Adversarial Network for Time Series Classification and Clustering
Recent works have demonstrated the superiority of supervised Convolutional Neural Networks (CNNs) in learning hierarchical representations from time series data for successful classification. These methods require sufficiently large labeled data for stable learning, however acquiring high-quality labeled time series data can be costly and potentially infeasible. Generative Adversarial Networks (GANs) have achieved great success in enhancing unsupervised and semi-supervised learning. Nonetheless, to our best knowledge, it remains unclear how effectively GANs can serve as a general-purpose solution to learn representations for time series recognition, i.e., classification and clustering. The above considerations inspire us to introduce a Time-series Convolutional GAN (TCGAN). TCGAN learns by playing an adversarial game between two one-dimensional CNNs (i.e., a generator and a discriminator) in the absence of label information. Parts of the trained TCGAN are then reused to construct a representation encoder to empower linear recognition methods. We conducted comprehensive experiments on synthetic and real-world datasets. The results demonstrate that TCGAN is faster and more accurate than existing time-series GANs. The learned representations enable simple classification and clustering methods to achieve superior and stable performance. Furthermore, TCGAN retains high efficacy in scenarios with few-labeled and imbalanced-labeled data. Our work provides a promising path to effectively utilize abundant unlabeled time series data.
Open AccessTCGAN: Convolutional Generative Adversarial Network for Time Series Classification and Clustering
Recent works have demonstrated the superiority of supervised Convolutional Neural Networks (CNNs) in learning hierarchical representations from time series data for successful classification. These methods require sufficiently large labeled data for stable learning, however acquiring high-quality labeled time series data can be costly and potentially infeasible. Generative Adversarial Networks (GANs) have achieved great success in enhancing unsupervised and semi-supervised learning. Nonetheless, to our best knowledge, it remains unclear how effectively GANs can serve as a general-purpose solution to learn representations for time series recognition, i.e., classification and clustering. The above considerations inspire us to introduce a Time-series Convolutional GAN (TCGAN). TCGAN learns by playing an adversarial game between two one-dimensional CNNs (i.e., a generator and a discriminator) in the absence of label information. Parts of the trained TCGAN are then reused to construct a representation encoder to empower linear recognition methods. We conducted comprehensive experiments on synthetic and real-world datasets. The results demonstrate that TCGAN is faster and more accurate than existing time-series GANs. The learned representations enable simple classification and clustering methods to achieve superior and stable performance. Furthermore, TCGAN retains high efficacy in scenarios with few-labeled and imbalanced-labeled data. Our work provides a promising path to effectively utilize abundant unlabeled time series data.
- Open Access
- 30 June 2023
- Published by Elsevier in Neural Networks
- Vol. 165, 868-883
TCGAN: Convolutional Generative Adversarial Network for time series classification and clustering
- 30 June 2023
- Published by Elsevier in Neural Networks
- Vol. 165, 868-883
Open AccessTCGAN: Convolutional Generative Adversarial Network for time series classification and clustering
- 5 May 2023
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 37  (4) , 1441-1472
A spatiotemporal deep neural network for fine-grained multi-horizon wind prediction
- 5 May 2023
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 37  (4) , 1441-1472
A spatiotemporal deep neural network for fine-grained multi-horizon wind prediction
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 7470-7483
Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 7470-7483
Open AccessBeat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
- 15 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 29  (23299290) , 3132-3143
Knowledge Enhanced Fact Checking and Verification
As the Internet and social media offer increasing opportunities for organizations and individuals to publicize online contents, it has become essential to develop effective means to identify misinformation like fake news. Recently, fact checking systems have been regarded as a promising tool to automatically deal with large amounts of information. How to effectively take advantage of existing unstructured document knowledge bases and structured knowledge graphs to build robust fact checking systems, however, remains to be a challenge. In this paper, we propose a knowledge enhanced fact checking system, which leverages the Wikidata5M knowledge graph and Wikipedia documents to incorporate external knowledge into the claim to be checked for more robust and accurate fact checking. First, we devise a contextualized knowledge graph selection method to identify the most relevant sub-graph with the checked claim from the large knowledge graph. We then construct a novel claim-evidence-knowledge graph and use a graph attention network to integrate natural language evidence with structured knowledge triplets by allowing them to propagate information among each other. By integrating the claim, retrieved evidence and selected knowledge triplets in a unified claim-evidence-knowledge graph, our method improves the label accuracy of predicted claims by more than 4% on the FEVER dataset over state-of-the-art fact checking models.
- 15 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 29  (23299290) , 3132-3143
Knowledge Enhanced Fact Checking and Verification
As the Internet and social media offer increasing opportunities for organizations and individuals to publicize online contents, it has become essential to develop effective means to identify misinformation like fake news. Recently, fact checking systems have been regarded as a promising tool to automatically deal with large amounts of information. How to effectively take advantage of existing unstructured document knowledge bases and structured knowledge graphs to build robust fact checking systems, however, remains to be a challenge. In this paper, we propose a knowledge enhanced fact checking system, which leverages the Wikidata5M knowledge graph and Wikipedia documents to incorporate external knowledge into the claim to be checked for more robust and accurate fact checking. First, we devise a contextualized knowledge graph selection method to identify the most relevant sub-graph with the checked claim from the large knowledge graph. We then construct a novel claim-evidence-knowledge graph and use a graph attention network to integrate natural language evidence with structured knowledge triplets by allowing them to propagate information among each other. By integrating the claim, retrieved evidence and selected knowledge triplets in a unified claim-evidence-knowledge graph, our method improves the label accuracy of predicted claims by more than 4% on the FEVER dataset over state-of-the-art fact checking models.
- Open Access
- 8 June 2021
- Published by Wiley in Mathematical Problems in Engineering
- Vol. 2021, 1-10
Catching Critical Transition in Engineered Systems
A variety of engineered systems can encounter critical transitions where the system suddenly shifts from one stable state to another at a critical threshold. The critical transition has aroused vital concerns for its potentially disastrous impacts. We validate an often taken-for-granted hypothesis that the failure of engineered systems can be attributed to the respective critical transitions and show how early warning signals are closely associated with critical transitions. We demonstrate that it is feasible to use early warning signals to predict system failures. Our findings open a new path to forecast failures of engineered systems with a generic method and provide supporting evidence for the universal existence of critical transition in dynamical systems at multiple scales.
- 8 June 2021
- Published by Wiley in Mathematical Problems in Engineering
- Vol. 2021, 1-10
Open AccessCatching Critical Transition in Engineered Systems
A variety of engineered systems can encounter critical transitions where the system suddenly shifts from one stable state to another at a critical threshold. The critical transition has aroused vital concerns for its potentially disastrous impacts. We validate an often taken-for-granted hypothesis that the failure of engineered systems can be attributed to the respective critical transitions and show how early warning signals are closely associated with critical transitions. We demonstrate that it is feasible to use early warning signals to predict system failures. Our findings open a new path to forecast failures of engineered systems with a generic method and provide supporting evidence for the universal existence of critical transition in dynamical systems at multiple scales.
- 28 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 32  (12) , 3066-3080
An Elastic Task Scheduling Scheme on Coarse-Grained Reconfigurable Architectures
Coarse-grained reconfigurable architectures (CGRAs) are increasingly employed as domain-specific accelerators due to their efficiency and flexibility. A CGRA typically relies on compilers to perform task scheduling. The longstanding problem of static scheduling is that it suffers from insufficient parallelism in handling irregularities due to over-serialization and workload imbalance, which leads to severe resource underutilization and performance loss. To counteract the limitations of static scheduling in CGRAs, it is essential to exploit dynamic parallelism automatically and manage hardware resources adaptively. However, existing dynamic scheduling mechanisms, e.g., work stealing, often reschedule aggressively for instant performance but sacrifice efficiency, which is unfavorable to CGRAs that emphasize efficiency and fewer reconfigurations. This paper proposes an elastic task scheduling scheme that enables lightweight dynamic scheduling in CGRAs. Tasks are rescheduled at runtime according to the classic tagged-token dataflow paradigm to enable dynamic task-level parallelism. Meanwhile, tasks are dynamically resized according to run-time throughputs via duplication, combination, and substitution operators for balanced multitask execution. We implement the elastic task scheduling scheme on a well-known reconfigurable architecture - triggered instruction architecture (TIA). Evaluation on the MachSuite benchmarks shows that the proposed scheme is effective in improving performance and energy efficiency. The average speedup is 2.2x over the baseline. Also, our design attains a 57% improvement in the area-normalized performance and a 49% better energy efficiency. Compared with a state-of-the-art dynamic scheduling method, our scheme achieves 1.6x speedup and 1.6x energy efficiency than work-stealing mechanism on the same substrate.
- 28 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 32  (12) , 3066-3080
An Elastic Task Scheduling Scheme on Coarse-Grained Reconfigurable Architectures
Coarse-grained reconfigurable architectures (CGRAs) are increasingly employed as domain-specific accelerators due to their efficiency and flexibility. A CGRA typically relies on compilers to perform task scheduling. The longstanding problem of static scheduling is that it suffers from insufficient parallelism in handling irregularities due to over-serialization and workload imbalance, which leads to severe resource underutilization and performance loss. To counteract the limitations of static scheduling in CGRAs, it is essential to exploit dynamic parallelism automatically and manage hardware resources adaptively. However, existing dynamic scheduling mechanisms, e.g., work stealing, often reschedule aggressively for instant performance but sacrifice efficiency, which is unfavorable to CGRAs that emphasize efficiency and fewer reconfigurations. This paper proposes an elastic task scheduling scheme that enables lightweight dynamic scheduling in CGRAs. Tasks are rescheduled at runtime according to the classic tagged-token dataflow paradigm to enable dynamic task-level parallelism. Meanwhile, tasks are dynamically resized according to run-time throughputs via duplication, combination, and substitution operators for balanced multitask execution. We implement the elastic task scheduling scheme on a well-known reconfigurable architecture - triggered instruction architecture (TIA). Evaluation on the MachSuite benchmarks shows that the proposed scheme is effective in improving performance and energy efficiency. The average speedup is 2.2x over the baseline. Also, our design attains a 57% improvement in the area-normalized performance and a 49% better energy efficiency. Compared with a state-of-the-art dynamic scheduling method, our scheme achieves 1.6x speedup and 1.6x energy efficiency than work-stealing mechanism on the same substrate.