Nguyen Bach
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Nguyen Bach has a total of 8 co-authors. Here are the top co-authors by number of shared publications:
- Fei Huang
DAMO Academy Alibaba Group China
2 shared publications Kewei Tu
School of Information Science and Technology Engineering Research Center of Intelligent Vision DAMO Academy Alibaba Group ShanghaiTech University Shanghai Imaging University of Chinese Academy of Sciences
2 shared publicationsTao Wang
School of Information Science and Technology Engineering Research Center of Intelligent Vision DAMO Academy Alibaba Group ShanghaiTech University Shanghai Imaging University of Chinese Academy of Sciences
2 shared publicationsXinyu Wang
School of Information Science and Technology Engineering Research Center of Intelligent Vision DAMO Academy Alibaba Group ShanghaiTech University Shanghai Imaging University of Chinese Academy of Sciences
2 shared publicationsYong Jiang
School of Information Science and Technology Engineering Research Center of Intelligent Vision DAMO Academy Alibaba Group ShanghaiTech University Shanghai Imaging University of Chinese Academy of Sciences
2 shared publicationsZhongqiang Huang
School of Information Science and Technology Engineering Research Center of Intelligent Vision DAMO Academy Alibaba Group ShanghaiTech University Shanghai Imaging University of Chinese Academy of Sciences
2 shared publications
Latest Publications
- Open Access
Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
Knowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge distillation is typically the cross-entropy between the teacher and the student's output distributions. However, for structured prediction problems, the output space is exponential in size; therefore, the cross-entropy objective becomes intractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, which is tractable for many typical choices of the teacher and student models. In particular, we show the tractability and empirical effectiveness of structural knowledge distillation between sequence labeling and dependency parsing models under four different scenarios: 1) the teacher and student share the same factorization form of the output structure scoring function; 2) the student factorization produces more fine-grained substructures than the teacher factorization; 3) the teacher factorization produces more fine-grained substructures than the student factorization; 4) the factorization forms from the teacher and the student are incompatible.
Open AccessStructural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
Knowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge distillation is typically the cross-entropy between the teacher and the student's output distributions. However, for structured prediction problems, the output space is exponential in size; therefore, the cross-entropy objective becomes intractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, which is tractable for many typical choices of the teacher and student models. In particular, we show the tractability and empirical effectiveness of structural knowledge distillation between sequence labeling and dependency parsing models under four different scenarios: 1) the teacher and student share the same factorization form of the output structure scoring function; 2) the student factorization produces more fine-grained substructures than the teacher factorization; 3) the teacher factorization produces more fine-grained substructures than the student factorization; 4) the factorization forms from the teacher and the student are incompatible.
- Open Access
More Embeddings, Better Sequence Labelers?
Recent work proposes a family of contextual embeddings that significantly improves the accuracy of sequence labelers over non-contextual embeddings. However, there is no definite conclusion on whether we can build better sequence labelers by combining different kinds of embeddings in various settings. In this paper, we conduct extensive experiments on 3 tasks over 18 datasets and 8 languages to study the accuracy of sequence labeling with various embedding concatenations and make three observations: (1) concatenating more embedding variants leads to better accuracy in rich-resource and cross-domain settings and some conditions of low-resource settings; (2) concatenating additional contextual sub-word embeddings with contextual character embeddings hurts the accuracy in extremely low-resource settings; (3) based on the conclusion of (1), concatenating additional similar contextual embeddings cannot lead to further improvements. We hope these conclusions can help people build stronger sequence labelers in various settings.
Open AccessMore Embeddings, Better Sequence Labelers?
Recent work proposes a family of contextual embeddings that significantly improves the accuracy of sequence labelers over non-contextual embeddings. However, there is no definite conclusion on whether we can build better sequence labelers by combining different kinds of embeddings in various settings. In this paper, we conduct extensive experiments on 3 tasks over 18 datasets and 8 languages to study the accuracy of sequence labeling with various embedding concatenations and make three observations: (1) concatenating more embedding variants leads to better accuracy in rich-resource and cross-domain settings and some conditions of low-resource settings; (2) concatenating additional contextual sub-word embeddings with contextual character embeddings hurts the accuracy in extremely low-resource settings; (3) based on the conclusion of (1), concatenating additional similar contextual embeddings cannot lead to further improvements. We hope these conclusions can help people build stronger sequence labelers in various settings.