Lu Yu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Lu Yu has a total of 30 co-authors. Here are the top co-authors by number of shared publications:
- Jiajun Zhang
School of Artificial Intelligence University of Chinese Academy of Sciences
2 shared publications - Wei Xiang
La Trobe University
2 shared publications Alex Gomez-Villa
1 shared publication- Andrew David Bagdanov
University of Florence University of Florence
1 shared publication - Arnau Ramisa
Amazon, Palo Alto, CA, USA
1 shared publication - Bartłomiej Twardowski
IDEAS NCBR
1 shared publication
Latest Publications
- Open Access
Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model
Recent research has demonstrated that the combination of pretrained diffusion models with neural radiance fields (NeRFs) has emerged as a promising approach for text-to-3D generation. Simply coupling NeRF with diffusion models will result in cross-view inconsistency and degradation of stylized view syntheses. To address this challenge, we propose the Edit-DiffNeRF framework, which is composed of a frozen diffusion model, a proposed delta module to edit the latent semantic space of the diffusion model, and a NeRF. Instead of training the entire diffusion for each scene, our method focuses on editing the latent semantic space in frozen pretrained diffusion models by the delta module. This fundamental change to the standard diffusion framework enables us to make fine-grained modifications to the rendered views and effectively consolidate these instructions in a 3D scene via NeRF training. As a result, we are able to produce an edited 3D scene that faithfully aligns to input text instructions. Furthermore, to ensure semantic consistency across different viewpoints, we propose a novel multi-view semantic consistency loss that extracts a latent semantic embedding from the input view as a prior, and aim to reconstruct it in different views. Our proposed method has been shown to effectively edit real-world 3D scenes, resulting in 25% improvement in the alignment of the performed 3D edits with text instructions compared to prior work.
Open AccessEdit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model
Recent research has demonstrated that the combination of pretrained diffusion models with neural radiance fields (NeRFs) has emerged as a promising approach for text-to-3D generation. Simply coupling NeRF with diffusion models will result in cross-view inconsistency and degradation of stylized view syntheses. To address this challenge, we propose the Edit-DiffNeRF framework, which is composed of a frozen diffusion model, a proposed delta module to edit the latent semantic space of the diffusion model, and a NeRF. Instead of training the entire diffusion for each scene, our method focuses on editing the latent semantic space in frozen pretrained diffusion models by the delta module. This fundamental change to the standard diffusion framework enables us to make fine-grained modifications to the rendered views and effectively consolidate these instructions in a 3D scene via NeRF training. As a result, we are able to produce an edited 3D scene that faithfully aligns to input text instructions. Furthermore, to ensure semantic consistency across different viewpoints, we propose a novel multi-view semantic consistency loss that extracts a latent semantic embedding from the input view as a prior, and aim to reconstruct it in different views. Our proposed method has been shown to effectively edit real-world 3D scenes, resulting in 25% improvement in the alignment of the performed 3D edits with text instructions compared to prior work.
- Open Access
X-Pruner: eXplainable Pruning for Vision Transformers
Recently vision transformer models have become prominent models for a range of tasks. These models, however, usually suffer from intensive computational costs and heavy memory requirements, making them impractical for deployment on edge platforms. Recent studies have proposed to prune transformers in an unexplainable manner, which overlook the relationship between internal units of the model and the target class, thereby leading to inferior performance. To alleviate this problem, we propose a novel explainable pruning framework dubbed X-Pruner, which is designed by considering the explainability of the pruning criterion. Specifically, to measure each prunable unit's contribution to predicting each target class, a novel explainability-aware mask is proposed and learned in an end-to-end manner. Then, to preserve the most informative units and learn the layer-wise pruning rate, we adaptively search the layer-wise threshold that differentiates between unpruned and pruned units based on their explainability-aware mask values. To verify and evaluate our method, we apply the X-Pruner on representative transformer models including the DeiT and Swin Transformer. Comprehensive simulation results demonstrate that the proposed X-Pruner outperforms the state-of-the-art black-box methods with significantly reduced computational costs and slight performance degradation.
Open AccessX-Pruner: eXplainable Pruning for Vision Transformers
Recently vision transformer models have become prominent models for a range of tasks. These models, however, usually suffer from intensive computational costs and heavy memory requirements, making them impractical for deployment on edge platforms. Recent studies have proposed to prune transformers in an unexplainable manner, which overlook the relationship between internal units of the model and the target class, thereby leading to inferior performance. To alleviate this problem, we propose a novel explainable pruning framework dubbed X-Pruner, which is designed by considering the explainability of the pruning criterion. Specifically, to measure each prunable unit's contribution to predicting each target class, a novel explainability-aware mask is proposed and learned in an end-to-end manner. Then, to preserve the most informative units and learn the layer-wise pruning rate, we adaptively search the layer-wise threshold that differentiates between unpruned and pruned units based on their explainability-aware mask values. To verify and evaluate our method, we apply the X-Pruner on representative transformer models including the DeiT and Swin Transformer. Comprehensive simulation results demonstrate that the proposed X-Pruner outperforms the state-of-the-art black-box methods with significantly reduced computational costs and slight performance degradation.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 2891-2907
Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only
Jinliang Lu, Yu Lu, Jiajun Zhang. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 2891-2907
Open AccessTake a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only
Jinliang Lu, Yu Lu, Jiajun Zhang. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 3422-3438
Adversarial Textual Robustness on Visual Dialog
Lu Yu, Verena Rieser. Findings of the Association for Computational Linguistics: ACL 2023. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 3422-3438
Open AccessAdversarial Textual Robustness on Visual Dialog
Lu Yu, Verena Rieser. Findings of the Association for Computational Linguistics: ACL 2023. 2023.
- 9 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 30 (23299290) , 1927-1938
Attention Analysis and Calibration for Transformer in Natural Language Generation
Attention mechanism has been ubiquitous in neural machine translation by dynamically selecting relevant contexts for different translations. Apart from performance gains, attention weights assigned to input tokens are often utilized to explain that high-attention tokens contribute more to the prediction. However, many works question whether this assumption holds in text classification by manually manipulating attention weights and observing decision flips. This article extends this question to Transformer-based neural machine translation, which heavily relies on cross-lingual attention to produce accurate translations but is relatively understudied in this context. We first design a mask perturbation model which automatically assesses each input’s contribution to model outputs. We then test whether the token contributing most to the current translation receives the highest attention weight. We find that it sometimes does not, which closely depends on the entropy of attention weights, the syntactic role of the current generation, and language pairs. We also rethink the discrepancy between attention weights and word alignments from the view of unreliable attention weights. Our observations further motivate us to calibrate the cross-lingual multi-head attention by attaching more attention to indispensable tokens, whose removal leads to a dramatic performance drop. Empirical experiments on different-scale translation tasks and text summarization tasks demonstrate that our calibration methods significantly outperform strong baselines.
- 9 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 30 (23299290) , 1927-1938
Attention Analysis and Calibration for Transformer in Natural Language Generation
Attention mechanism has been ubiquitous in neural machine translation by dynamically selecting relevant contexts for different translations. Apart from performance gains, attention weights assigned to input tokens are often utilized to explain that high-attention tokens contribute more to the prediction. However, many works question whether this assumption holds in text classification by manually manipulating attention weights and observing decision flips. This article extends this question to Transformer-based neural machine translation, which heavily relies on cross-lingual attention to produce accurate translations but is relatively understudied in this context. We first design a mask perturbation model which automatically assesses each input’s contribution to model outputs. We then test whether the token contributing most to the current translation receives the highest attention weight. We find that it sometimes does not, which closely depends on the entropy of attention weights, the syntactic role of the current generation, and language pairs. We also rethink the discrepancy between attention weights and word alignments from the view of unreliable attention weights. Our observations further motivate us to calibrate the cross-lingual multi-head attention by attaching more attention to indispensable tokens, whose removal leads to a dramatic performance drop. Empirical experiments on different-scale translation tasks and text summarization tasks demonstrate that our calibration methods significantly outperform strong baselines.
- Open Access
Continually Learning Self-Supervised Representations with Projected Functional Regularization
Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay mechanism. We show that naive functional regularization, also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in different scenarios and on multiple datasets.
Open AccessContinually Learning Self-Supervised Representations with Projected Functional Regularization
Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay mechanism. We show that naive functional regularization, also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in different scenarios and on multiple datasets.
- Open Access
Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance
We study stochastic convex optimization under infinite noise variance. Specifically, when the stochastic gradient is unbiased and has uniformly bounded $(1+\kappa)$-th moment, for some $\kappa \in (0,1]$, we quantify the convergence rate of the Stochastic Mirror Descent algorithm with a particular class of uniformly convex mirror maps, in terms of the number of iterations, dimensionality and related geometric parameters of the optimization problem. Interestingly this algorithm does not require any explicit gradient clipping or normalization, which have been extensively used in several recent empirical and theoretical works. We complement our convergence results with information-theoretic lower bounds showing that no other algorithm using only stochastic first-order oracles can achieve improved rates. Our results have several interesting consequences for devising online/streaming stochastic approximation algorithms for problems arising in robust statistics and machine learning.Open AccessMirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance
We study stochastic convex optimization under infinite noise variance. Specifically, when the stochastic gradient is unbiased and has uniformly bounded $(1+\kappa)$-th moment, for some $\kappa \in (0,1]$, we quantify the convergence rate of the Stochastic Mirror Descent algorithm with a particular class of uniformly convex mirror maps, in terms of the number of iterations, dimensionality and related geometric parameters of the optimization problem. Interestingly this algorithm does not require any explicit gradient clipping or normalization, which have been extensively used in several recent empirical and theoretical works. We complement our convergence results with information-theoretic lower bounds showing that no other algorithm using only stochastic first-order oracles can achieve improved rates. Our results have several interesting consequences for devising online/streaming stochastic approximation algorithms for problems arising in robust statistics and machine learning. - Open Access
Learning Metrics from Teachers: Compact Networks for Image Embedding
Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)
Open AccessLearning Metrics from Teachers: Compact Networks for Image Embedding
Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)
- 9 January 2019
- Published by Springer Nature
- p. 27-38
Exploiting Knowledge Graph in Neural Machine Translation
- 9 January 2019
- Published by Springer Nature
- p. 27-38
Exploiting Knowledge Graph in Neural Machine Translation
- 12 May 2011
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Oceanic Engineering
- Vol. 36 (2) , 273-284
Performance Analysis of Matched-Field Source Localization Under Spatially Correlated Noise Field
Matched-field processing (MFP) concerns estimation of source locations by exploiting full wave modeling of acoustic waveguide propagation. Typical MFP performance demonstrates a threshold behavior, that is, below some signal-to-noise ratio (SNR), the mean-square error (MSE) increases dramatically. This threshold phenomenon has been intensively investigated previously under spatially white noise field. In this paper, we develop approaches to study MFP performance in the presence of spatially correlated noises. Both the discrete interference and surface-generated noise are considered, and a so-called method of intervals errors (MIE) is derived in the context of the maximum likelihood estimate (MLE). Performance analysis of both narrowband and broadband MFP is implemented for source localization in a realistic shallow water environment. Simulation results suggest that 1) adding either spatially correlated noise increases the MLE threshold SNR compared to the case with white noise only, but effects at high SNR are different, MSE decreasing at the same rate with a point interference while decaying in a slower pace with surface-generated noises; 2) with a strong discrete interference, interference position, rather than strength, matters more in determining the MLE performance, and broadband processing effectively reduces the threshold SNR; 3) effect of the surface-generated noises is highly frequency dependent, and processing using a higher frequency would be preferred; 4) overall the surface-generated noise is more difficult to attack by an adaptive method such as the MLE.
- 12 May 2011
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Oceanic Engineering
- Vol. 36 (2) , 273-284
Performance Analysis of Matched-Field Source Localization Under Spatially Correlated Noise Field
Matched-field processing (MFP) concerns estimation of source locations by exploiting full wave modeling of acoustic waveguide propagation. Typical MFP performance demonstrates a threshold behavior, that is, below some signal-to-noise ratio (SNR), the mean-square error (MSE) increases dramatically. This threshold phenomenon has been intensively investigated previously under spatially white noise field. In this paper, we develop approaches to study MFP performance in the presence of spatially correlated noises. Both the discrete interference and surface-generated noise are considered, and a so-called method of intervals errors (MIE) is derived in the context of the maximum likelihood estimate (MLE). Performance analysis of both narrowband and broadband MFP is implemented for source localization in a realistic shallow water environment. Simulation results suggest that 1) adding either spatially correlated noise increases the MLE threshold SNR compared to the case with white noise only, but effects at high SNR are different, MSE decreasing at the same rate with a point interference while decaying in a slower pace with surface-generated noises; 2) with a strong discrete interference, interference position, rather than strength, matters more in determining the MLE performance, and broadband processing effectively reduces the threshold SNR; 3) effect of the surface-generated noises is highly frequency dependent, and processing using a higher frequency would be preferred; 4) overall the surface-generated noise is more difficult to attack by an adaptive method such as the MLE.