LYLu Yu

Lu Yu

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Lu Yu has a total of 30 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model

    by Lu Yu, Wei Xiang, Kang Han

    Recent research has demonstrated that the combination of pretrained diffusion models with neural radiance fields (NeRFs) has emerged as a promising approach for text-to-3D generation. Simply coupling NeRF with diffusion models will result in cross-view inconsistency and degradation of stylized view syntheses. To address this challenge, we propose the Edit-DiffNeRF framework, which is composed of a frozen diffusion model, a proposed delta module to edit the latent semantic space of the diffusion model, and a NeRF. Instead of training the entire diffusion for each scene, our method focuses on editing the latent semantic space in frozen pretrained diffusion models by the delta module. This fundamental change to the standard diffusion framework enables us to make fine-grained modifications to the rendered views and effectively consolidate these instructions in a 3D scene via NeRF training. As a result, we are able to produce an edited 3D scene that faithfully aligns to input text instructions. Furthermore, to ensure semantic consistency across different viewpoints, we propose a novel multi-view semantic consistency loss that extracts a latent semantic embedding from the input view as a prior, and aim to reconstruct it in different views. Our proposed method has been shown to effectively edit real-world 3D scenes, resulting in 25% improvement in the alignment of the performed 3D edits with text instructions compared to prior work.

  • Open Access

    X-Pruner: eXplainable Pruning for Vision Transformers

    by Lu Yu, Wei Xiang

    Recently vision transformer models have become prominent models for a range of tasks. These models, however, usually suffer from intensive computational costs and heavy memory requirements, making them impractical for deployment on edge platforms. Recent studies have proposed to prune transformers in an unexplainable manner, which overlook the relationship between internal units of the model and the target class, thereby leading to inferior performance. To alleviate this problem, we propose a novel explainable pruning framework dubbed X-Pruner, which is designed by considering the explainability of the pruning criterion. Specifically, to measure each prunable unit's contribution to predicting each target class, a novel explainability-aware mask is proposed and learned in an end-to-end manner. Then, to preserve the most informative units and learn the layer-wise pruning rate, we adaptively search the layer-wise threshold that differentiates between unpruned and pruned units based on their explainability-aware mask values. To verify and evaluate our method, we apply the X-Pruner on representative transformer models including the DeiT and Swin Transformer. Comprehensive simulation results demonstrate that the proposed X-Pruner outperforms the state-of-the-art black-box methods with significantly reduced computational costs and slight performance degradation.

  • Open Access

    Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only

    by Jinliang Lu, Yu Lu, Jiajun Zhang

    Jinliang Lu, Yu Lu, Jiajun Zhang. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.


  • Open Access

    Adversarial Textual Robustness on Visual Dialog

    by Lu Yu, Verena Rieser

    Lu Yu, Verena Rieser. Findings of the Association for Computational Linguistics: ACL 2023. 2023.


  • Attention Analysis and Calibration for Transformer in Natural Language Generation

    by Yu Lu, Jiajun Zhang, Jiali Zeng, Shuangzhi Wu, Chengqing Zong

    Attention mechanism has been ubiquitous in neural machine translation by dynamically selecting relevant contexts for different translations. Apart from performance gains, attention weights assigned to input tokens are often utilized to explain that high-attention tokens contribute more to the prediction. However, many works question whether this assumption holds in text classification by manually manipulating attention weights and observing decision flips. This article extends this question to Transformer-based neural machine translation, which heavily relies on cross-lingual attention to produce accurate translations but is relatively understudied in this context. We first design a mask perturbation model which automatically assesses each input’s contribution to model outputs. We then test whether the token contributing most to the current translation receives the highest attention weight. We find that it sometimes does not, which closely depends on the entropy of attention weights, the syntactic role of the current generation, and language pairs. We also rethink the discrepancy between attention weights and word alignments from the view of unreliable attention weights. Our observations further motivate us to calibrate the cross-lingual multi-head attention by attaching more attention to indispensable tokens, whose removal leads to a dramatic performance drop. Empirical experiments on different-scale translation tasks and text summarization tasks demonstrate that our calibration methods significantly outperform strong baselines.


  • Open Access

    Continually Learning Self-Supervised Representations with Projected Functional Regularization

    by Alex Gomez-Villa, Bartlomiej Twardowski, Lu Yu, Andrew D. Bagdanov, Joost van de Weijer

    Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay mechanism. We show that naive functional regularization, also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in different scenarios and on multiple datasets.

  • Open Access

    Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance

    by Nuri Mert Vural, Lu Yu, Krishnakumar Balasubramanian, Stanislav Volgushev, Murat A. Erdogdu

    We study stochastic convex optimization under infinite noise variance. Specifically, when the stochastic gradient is unbiased and has uniformly bounded $(1+\kappa)$-th moment, for some $\kappa \in (0,1]$, we quantify the convergence rate of the Stochastic Mirror Descent algorithm with a particular class of uniformly convex mirror maps, in terms of the number of iterations, dimensionality and related geometric parameters of the optimization problem. Interestingly this algorithm does not require any explicit gradient clipping or normalization, which have been extensively used in several recent empirical and theoretical works. We complement our convergence results with information-theoretic lower bounds showing that no other algorithm using only stochastic first-order oracles can achieve improved rates. Our results have several interesting consequences for devising online/streaming stochastic approximation algorithms for problems arising in robust statistics and machine learning.

  • Open Access

    Learning Metrics from Teachers: Compact Networks for Image Embedding

    by Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, Arnau Ramisa

    Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)

  • Performance Analysis of Matched-Field Source Localization Under Spatially Correlated Noise Field

    by Wen Xu, Zhuan Xiao, Lu Yu

    Matched-field processing (MFP) concerns estimation of source locations by exploiting full wave modeling of acoustic waveguide propagation. Typical MFP performance demonstrates a threshold behavior, that is, below some signal-to-noise ratio (SNR), the mean-square error (MSE) increases dramatically. This threshold phenomenon has been intensively investigated previously under spatially white noise field. In this paper, we develop approaches to study MFP performance in the presence of spatially correlated noises. Both the discrete interference and surface-generated noise are considered, and a so-called method of intervals errors (MIE) is derived in the context of the maximum likelihood estimate (MLE). Performance analysis of both narrowband and broadband MFP is implemented for source localization in a realistic shallow water environment. Simulation results suggest that 1) adding either spatially correlated noise increases the MLE threshold SNR compared to the case with white noise only, but effects at high SNR are different, MSE decreasing at the same rate with a point interference while decaying in a slower pace with surface-generated noises; 2) with a strong discrete interference, interference position, rather than strength, matters more in determining the MLE performance, and broadband processing effectively reduces the threshold SNR; 3) effect of the surface-generated noises is highly frequency dependent, and processing using a higher frequency would be preferred; 4) overall the surface-generated noise is more difficult to attack by an adaptive method such as the MLE.