Jiwen Lu

In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM without fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. However, the Attention Gravity and Attention Sink phenomena undermine the sparsification reference of SparseVLM: text-visual alignment. To address this, we introduce improved text-visual attention patterns to enhance the fidelity of query-aware vision token selection. First, we correct the Attention Gravity effect, where positional encodings induce disproportionately high attention to vision tokens positioned near textual tokens. We further mitigate the Attention Sink by selecting priority attention heads that suppress the excessive attention absorbed by semantically irrelevant vision tokens. The two parts refine the text-visual alignment from both the token and head perspectives. Finally, for video understanding tasks, we propose Temporal-Balanced Sparsification, complementing spatial relevance with temporal diversity to reduce cross-frame redundancy. Extensive experiments on 23 widely used image and video understanding tasks across 7 vision-language models demonstrate the effectiveness of our SparseVLM+ framework. For example, equipping LLaVA with SparseVLM+ yields a 57.1% reduction in FLOPs and a 38.4% decrease in CUDA latency, while preserving 99.6% of its original accuracy on average. Our code is available at https://github.com/Gumpest/SparseVLMs.

Point cloud completion aims to reconstruct the geometry of partial point clouds captured by various sensors. Traditionally, point cloud models are trained on synthetic datasets that feature limited categories and differ significantly from real-world scenarios. This gap often causes existing methods to struggle when faced with unfamiliar categories and severe incompleteness in real-world applications. In this paper, we propose PrototypeCompletion, a novel prototype-based approach for point cloud completion. The method begins by generating rough prototypes, which are then refined with additional geometric details to make the final prediction. We introduce two distinct approaches for integrating prototypes into the network: explicit prototypes and implicit prototypes. Our approach demonstrates strong generalization capabilities, allowing it to handle point cloud completion for a variety of unseen categories beyond the training data. We demonstrate that incorporating language prompts into the training of point cloud completion models significantly expands their applicability and enhances their performance in diverse point cloud completion tasks. Furthermore, we propose a new evaluation metric and a test benchmark based on ScanNet200 and KITTI, designed to assess the model's performance in real-world scenarios and foster future research in the field. Experimental results show that our method outperforms state-of-the-art models on the existing PCN and ShapeNet34 benchmarks and also excels in various real-world settings, handling different object categories and sensor types effectively. The code will be made publicly available.

Deep learning has profoundly impacted society, yet the inherent nature of deep neural networks hinders further application to high-reliability industries. To demystify these closed-boxes, numerous works attempt to improve the explainability by observing or impacting internal variables of the models. However, existing methods rely on heuristics without rigorous theoretical foundations, often requiring intricate model modifications or redesigns. This work first formalizes two fundamental properties of explainability: alignment and invertibility, serving as theoretical pillars for rigorous interpretability analysis. Building on these, we introduce Bort, a plug-and-play optimizer that enforces Boundedness and orthogonality constraints on model parameters to improve explainability. These constraints are theoretically derived from the alignment and invertibility principles. Considering conventional optimizers can not leverage data features for precise attribution, we present a data-aware extension, termed DBort, which integrates an auxiliary loss term. Intriguingly, in the linear case, DBort converges to Principal Component Analysis (PCA). Our in-depth analysis of penalty term design reveals that $l_{1}$l1-based penalties provide a more stringent adherence to the imposed constraints compared to their $l_{2}$l2 counterparts. Our experiments involve reconstructing and backtracking through the optimized model representations, which reveal a marked enhancement in explainability. Furthermore, leveraging Bort, we successfully synthesize explainable adversarial examples without additional training. Notably, Bort consistently improves the classification accuracy across diverse architectures, including ResNet and DeiT, on benchmark datasets such as MNIST, CIFAR-10, and ImageNet.

Building on the. success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc. Besides, we propose a new multi-stage refinement technique that is designed to reduce the inference costs with large flow-based image generation models. Specifically, the multi-stage refinement split the whole generation procedure on different resolutions, forming a coarse-to-fine text-to-image pipeline. We further adopt a stage-aware deployment strategy that can maximize the inference speed in terms of both latency and throughput. By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1%$\sim$∼58.3% on class-conditional generation and 29.8%$\sim$∼38.5% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms/img) and FID of 3.93 with 38 (ms/img), achieving the real-time image generation and establishing the new state-of-the-art. Equipped with the recent SD 3.5 Large, we achieved FID of 28.05 with a speed improvement of around 50% on NVIDIA 3090 GPU.