Zongru Wu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Zongru Wu has a total of 22 co-authors. Here are the top co-authors by number of shared publications:
- Zhuosheng Zhang
Shanghai Jiao Tong University, Shanghai, China
6 shared publications Gongshen Liu
School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University
4 shared publicationsPengzhou Cheng
School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University
4 shared publications- Gongshen Liu
Shanghai Jiao Tong University, Shanghai, China
3 shared publications - Pengzhou Cheng
Shanghai Jiao Tong University
3 shared publications Tianjie Ju
School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University
2 shared publications
Latest Publications
- 29 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 3290-3302
Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-Based Mobile-Using Agents
Recent advancements in multimodal large language models (MLLMs) have shown exceptional potential in enabling mobile-using agents to autonomously execute human instructions. However, fully automated agents often try to execute tasks even when they are unable to resolve them, leading to the problem of over-execution. Previous studies solve it by training a interactive mobile-using agents to let agents request human interaction when agents can not complete user instructions. However, we find that these interactive agents tend to exhibit over-soliciting behavior, relying excessively on human intervention. To mitigate both over-execution and over-soliciting, we propose a universal confidence integration framework that enables confidence-driven proactive and robust interaction in MLLM-based mobile-using agents. The framework consists of two stages: interaction capability empowerment and confidence bias correction. In the interaction capability empowerment stage, agents learn through supervised fine-tuning to output both actions and confidence scores. In the confidence bias correction stage, agents learn to output more accurate confidence scores by combining semantic similarity retrieval with direct preference optimization. Experimental results show Mobile-Aptus achieves state-of-the-art performance on the four popular mobile-using agent benchmarks: OS-Kairos, AITZ, Meta-GUI, and AndroidControl. Mobile-Aptus consistently outperforms all baselines in offline benchmarks, with an average improvement over 17% in task success rate. In real-world dynamic experiments, Mobile-Aptus surpasses the baseline by 26% in task success rate with only 0.64 intervention steps per instruction. Moreover, we find that the salient human interventions can be effectively replaced by a multi-agent system to avoid task interruption. The codes are available at https://github.com/Wuzheng02/Mobile-Aptus.
- 29 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 3290-3302
Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-Based Mobile-Using Agents
Recent advancements in multimodal large language models (MLLMs) have shown exceptional potential in enabling mobile-using agents to autonomously execute human instructions. However, fully automated agents often try to execute tasks even when they are unable to resolve them, leading to the problem of over-execution. Previous studies solve it by training a interactive mobile-using agents to let agents request human interaction when agents can not complete user instructions. However, we find that these interactive agents tend to exhibit over-soliciting behavior, relying excessively on human intervention. To mitigate both over-execution and over-soliciting, we propose a universal confidence integration framework that enables confidence-driven proactive and robust interaction in MLLM-based mobile-using agents. The framework consists of two stages: interaction capability empowerment and confidence bias correction. In the interaction capability empowerment stage, agents learn through supervised fine-tuning to output both actions and confidence scores. In the confidence bias correction stage, agents learn to output more accurate confidence scores by combining semantic similarity retrieval with direct preference optimization. Experimental results show Mobile-Aptus achieves state-of-the-art performance on the four popular mobile-using agent benchmarks: OS-Kairos, AITZ, Meta-GUI, and AndroidControl. Mobile-Aptus consistently outperforms all baselines in offline benchmarks, with an average improvement over 17% in task success rate. In real-world dynamic experiments, Mobile-Aptus surpasses the baseline by 26% in task success rate with only 0.64 intervention steps per instruction. Moreover, we find that the salient human interventions can be effectively replaced by a multi-agent system to avoid task interruption. The codes are available at https://github.com/Wuzheng02/Mobile-Aptus.
- Open Access
- 17 March 2026
- Published by Elsevier in SSRN Electronic Journal
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Task
Domain adaptation, particularly through grounding techniques, is widely adopted for multimodal large language models (MLLMs) to enhance the perception of graphical user interface (GUI) environments, further improving the reasoning performance of MLLM-powered GUI agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and goal-oriented reasoning limits the adaptation effectiveness. To investigate a supplementary domain adaptation method that can further improve GUI perception and reasoning, we propose a query-oriented pivot approach called Intended Query Inference (InQuIre). From an information-theoretic perspective, InQuIre serves as a semantic pivot task that maximizes the mutual information between visual representations and user intent. By explicitly deducing the intended queries behind action coordinates, InQuIre provides dense supervision to align coordinate-indicated visual features with functional intent, offering superior data efficiency compared to sparse coordinate supervision. Experimental results focusing on mobile platform show that InQuIre outperforms grounding under the same adaptation data scale. Notably, InQuIre achieves comparable performance to large-scale grounding-adapted OS-Atlas with less than 0.1% of adaptation data. Furthermore, through experiments in cross-platform scenarios, InQuIre demonstrates its generalizability. Additionally, we explore the impact of reasoning formats and highlight that integrating additional semantic information into the input further boosts reasoning performance. Finally, through attention visualization, we validate that InQuIre effectively guides the focus toward functionally relevant GUI elements, ensuring robust comprehension of user intent. The code is publicly available at https://github.com/ZrW00/GUIPivot.
- 17 March 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessSmoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Task
Domain adaptation, particularly through grounding techniques, is widely adopted for multimodal large language models (MLLMs) to enhance the perception of graphical user interface (GUI) environments, further improving the reasoning performance of MLLM-powered GUI agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and goal-oriented reasoning limits the adaptation effectiveness. To investigate a supplementary domain adaptation method that can further improve GUI perception and reasoning, we propose a query-oriented pivot approach called Intended Query Inference (InQuIre). From an information-theoretic perspective, InQuIre serves as a semantic pivot task that maximizes the mutual information between visual representations and user intent. By explicitly deducing the intended queries behind action coordinates, InQuIre provides dense supervision to align coordinate-indicated visual features with functional intent, offering superior data efficiency compared to sparse coordinate supervision. Experimental results focusing on mobile platform show that InQuIre outperforms grounding under the same adaptation data scale. Notably, InQuIre achieves comparable performance to large-scale grounding-adapted OS-Atlas with less than 0.1% of adaptation data. Furthermore, through experiments in cross-platform scenarios, InQuIre demonstrates its generalizability. Additionally, we explore the impact of reasoning formats and highlight that integrating additional semantic information into the input further boosts reasoning performance. Finally, through attention visualization, we validate that InQuIre effectively guides the focus toward functionally relevant GUI elements, ensuring robust comprehension of user intent. The code is publicly available at https://github.com/ZrW00/GUIPivot.
- Open Access
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
Perception-enhanced pre-training, particularly through grounding techniques, is widely adopted to enhance the performance of graphical user interface (GUI) agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and action-oriented reasoning limits the effectiveness of grounding for reasoning tasks. To address this challenge, we propose a query-oriented pivot approach called query inference, which serves as a bridge between GUI grounding and reasoning. By inferring potential user queries from a screenshot and its associated element coordinates, query inference improves the understanding of coordinates while aligning more closely with reasoning tasks. Experimental results show that query inference outperforms previous grounding techniques under the same training data scale. Notably, query inference achieves comparable or even better performance to large-scale grounding-enhanced OS-Atlas with less than 0.1% of training data. Furthermore, we explore the impact of reasoning formats and demonstrate that integrating additional semantic information into the input further boosts reasoning performance. The code is publicly available at https://github.com/ZrW00/GUIPivot.
Open AccessSmoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
Perception-enhanced pre-training, particularly through grounding techniques, is widely adopted to enhance the performance of graphical user interface (GUI) agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and action-oriented reasoning limits the effectiveness of grounding for reasoning tasks. To address this challenge, we propose a query-oriented pivot approach called query inference, which serves as a bridge between GUI grounding and reasoning. By inferring potential user queries from a screenshot and its associated element coordinates, query inference improves the understanding of coordinates while aligning more closely with reasoning tasks. Experimental results show that query inference outperforms previous grounding techniques under the same training data scale. Notably, query inference achieves comparable or even better performance to large-scale grounding-enhanced OS-Atlas with less than 0.1% of training data. Furthermore, we explore the impact of reasoning formats and demonstrate that integrating additional semantic information into the input further boosts reasoning performance. The code is publicly available at https://github.com/ZrW00/GUIPivot.
- Top Cited Papers
- 26 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (8) , 13628-13648
Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review
Language models (LMs) are becoming increasingly popular in real-world applications. Outsourcing model training and data hosting to third-party platforms has become a standard method for reducing costs. In such a situation, the attacker can manipulate the training process or data to inject a backdoor into models. Backdoor attacks are a serious threat where malicious behavior is activated when triggers are present; otherwise, the model operates normally. However, there is still no systematic and comprehensive review of LMs from the attacker’s capabilities and purposes on different backdoor attack surfaces. Moreover, there is a shortage of analysis and comparison of the diverse emerging backdoor countermeasures. Therefore, this work aims to provide the natural language processing (NLP) community with a timely review of backdoor attacks and countermeasures. According to the attackers’ capability and affected stage of the LMs, the attack surfaces are formalized into four categorizations: attacking the pretrained model with fine-tuning (APMF) or parameter-efficient fine-tuning (PEFT), attacking the final model with training (AFMT), and attacking large language model (ALLM). Thus, attacks under each categorization are combed. The countermeasures are categorized into two general classes: sample inspection and model inspection. Thus, we review countermeasures and analyze their advantages and disadvantages. Also, we summarize the benchmark datasets and provide comparable evaluations for representative attacks and defenses. Drawing the insights from the review, we point out the crucial areas for future research on the backdoor, especially soliciting more efficient and practical countermeasures.
- 26 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (8) , 13628-13648
Top Cited PapersBackdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review
Language models (LMs) are becoming increasingly popular in real-world applications. Outsourcing model training and data hosting to third-party platforms has become a standard method for reducing costs. In such a situation, the attacker can manipulate the training process or data to inject a backdoor into models. Backdoor attacks are a serious threat where malicious behavior is activated when triggers are present; otherwise, the model operates normally. However, there is still no systematic and comprehensive review of LMs from the attacker’s capabilities and purposes on different backdoor attack surfaces. Moreover, there is a shortage of analysis and comparison of the diverse emerging backdoor countermeasures. Therefore, this work aims to provide the natural language processing (NLP) community with a timely review of backdoor attacks and countermeasures. According to the attackers’ capability and affected stage of the LMs, the attack surfaces are formalized into four categorizations: attacking the pretrained model with fine-tuning (APMF) or parameter-efficient fine-tuning (PEFT), attacking the final model with training (AFMT), and attacking large language model (ALLM). Thus, attacks under each categorization are combed. The countermeasures are categorized into two general classes: sample inspection and model inspection. Thus, we review countermeasures and analyze their advantages and disadvantages. Also, we summarize the benchmark datasets and provide comparable evaluations for representative attacks and defenses. Drawing the insights from the review, we point out the crucial areas for future research on the backdoor, especially soliciting more efficient and practical countermeasures.
- 20 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 12 (11) , 16171-16180
Transferable and Robust Dynamic Adversarial Attack Against Object Detection Models
Object detection models have been widely deployed in physical world applications, and they are vulnerable to adversarial attacks. However, most adversarial attacks are implemented in a white-box setting, and under ideal shooting conditions such as fixed distances and angles, and thus have limited attack success rate (ASR) in practice. In this paper, we present a transferable and robust dynamic adversarial attack where the adversarial patches can be printed on or attached to non-rigid objects such as clothes. We develop a cascade module with a momentum-based technique to optimize adversarial patches against various object detection models, achieving better transferability of the patches in a black-box setting. We also develop a strategy of distance-adaptive patch generation and employ perspective transformation to enhance the robustness of patches. To evaluate the attack performance, we conduct extensive experiments on seven mainstream object detection models at different distances and angles. The results show that our method can achieve an average ASR of 69.85%, which is 3.27 times that of the baseline method at 3 meters.
- 20 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 12 (11) , 16171-16180
Transferable and Robust Dynamic Adversarial Attack Against Object Detection Models
Object detection models have been widely deployed in physical world applications, and they are vulnerable to adversarial attacks. However, most adversarial attacks are implemented in a white-box setting, and under ideal shooting conditions such as fixed distances and angles, and thus have limited attack success rate (ASR) in practice. In this paper, we present a transferable and robust dynamic adversarial attack where the adversarial patches can be printed on or attached to non-rigid objects such as clothes. We develop a cascade module with a momentum-based technique to optimize adversarial patches against various object detection models, achieving better transferability of the patches in a black-box setting. We also develop a strategy of distance-adaptive patch generation and employ perspective transformation to enhance the robustness of patches. To evaluate the attack performance, we conduct extensive experiments on seven mainstream object detection models at different distances and angles. The results show that our method can achieve an average ASR of 69.85%, which is 3.27 times that of the baseline method at 3 meters.
- Open Access
Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retraining
Backdoor attacks remain significant security threats to generative large language models (LLMs). Since generative LLMs output sequences of high-dimensional token logits instead of low-dimensional classification logits, most existing backdoor defense methods designed for discriminative models like BERT are ineffective for generative LLMs. Inspired by the observed differences in learning behavior between backdoor and clean mapping in the frequency space, we transform gradients of each training sample, directly influencing parameter updates, into the frequency space. Our findings reveal a distinct separation between the gradients of backdoor and clean samples in the frequency space. Based on this phenomenon, we propose Gradient Clustering in the Frequency Space for Backdoor Sample Filtering (GraCeFul), which leverages sample-wise gradients in the frequency space to effectively identify backdoor samples without requiring retraining LLMs. Experimental results show that GraCeFul outperforms baselines significantly. Notably, GraCeFul exhibits remarkable computational efficiency, achieving nearly 100% recall and F1 scores in identifying backdoor samples, reducing the average success rate of various backdoor attacks to 0% with negligible drops in clean accuracy across multiple free-style question answering datasets. Additionally, GraCeFul generalizes to Llama-2 and Vicuna. The codes are publicly available at https://github.com/ZrW00/GraceFul.
Open AccessGracefully Filtering Backdoor Samples for Generative Large Language Models without Retraining
Backdoor attacks remain significant security threats to generative large language models (LLMs). Since generative LLMs output sequences of high-dimensional token logits instead of low-dimensional classification logits, most existing backdoor defense methods designed for discriminative models like BERT are ineffective for generative LLMs. Inspired by the observed differences in learning behavior between backdoor and clean mapping in the frequency space, we transform gradients of each training sample, directly influencing parameter updates, into the frequency space. Our findings reveal a distinct separation between the gradients of backdoor and clean samples in the frequency space. Based on this phenomenon, we propose Gradient Clustering in the Frequency Space for Backdoor Sample Filtering (GraCeFul), which leverages sample-wise gradients in the frequency space to effectively identify backdoor samples without requiring retraining LLMs. Experimental results show that GraCeFul outperforms baselines significantly. Notably, GraCeFul exhibits remarkable computational efficiency, achieving nearly 100% recall and F1 scores in identifying backdoor samples, reducing the average success rate of various backdoor attacks to 0% with negligible drops in clean accuracy across multiple free-style question answering datasets. Additionally, GraCeFul generalizes to Llama-2 and Vicuna. The codes are publicly available at https://github.com/ZrW00/GraceFul.
- Open Access
Transferring Backdoors between Large Language Models by Knowledge Distillation
Backdoor Attacks have been a serious vulnerability against Large Language Models (LLMs). However, previous methods only reveal such risk in specific models, or present tasks transferability after attacking the pre-trained phase. So, how risky is the model transferability of a backdoor attack? In this paper, we focus on whether existing mini-LLMs may be unconsciously instructed in backdoor knowledge by poisoned teacher LLMs through knowledge distillation (KD). Specifically, we propose ATBA, an adaptive transferable backdoor attack, which can effectively distill the backdoor of teacher LLMs into small models when only executing clean-tuning. We first propose the Target Trigger Generation (TTG) module that filters out a set of indicative trigger candidates from the token list based on cosine similarity distribution. Then, we exploit a shadow model to imitate the distilling process and introduce an Adaptive Trigger Optimization (ATO) module to realize a gradient-based greedy feedback to search optimal triggers. Extensive experiments show that ATBA generates not only positive guidance for student models but also implicitly transfers backdoor knowledge. Our attack is robust and stealthy, with over 80% backdoor transferability, and hopes the attention of security.
Open AccessTransferring Backdoors between Large Language Models by Knowledge Distillation
Backdoor Attacks have been a serious vulnerability against Large Language Models (LLMs). However, previous methods only reveal such risk in specific models, or present tasks transferability after attacking the pre-trained phase. So, how risky is the model transferability of a backdoor attack? In this paper, we focus on whether existing mini-LLMs may be unconsciously instructed in backdoor knowledge by poisoned teacher LLMs through knowledge distillation (KD). Specifically, we propose ATBA, an adaptive transferable backdoor attack, which can effectively distill the backdoor of teacher LLMs into small models when only executing clean-tuning. We first propose the Target Trigger Generation (TTG) module that filters out a set of indicative trigger candidates from the token list based on cosine similarity distribution. Then, we exploit a shadow model to imitate the distilling process and introduce an Adaptive Trigger Optimization (ATO) module to realize a gradient-based greedy feedback to search optimal triggers. Extensive experiments show that ATBA generates not only positive guidance for student models but also implicitly transfers backdoor knowledge. Our attack is robust and stealthy, with over 80% backdoor transferability, and hopes the attention of security.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 8116-8134
Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space
Zongru Wu, Zhuosheng Zhang, Pengzhou Cheng, Gongshen Liu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 8116-8134
Open AccessAcquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space
Zongru Wu, Zhuosheng Zhang, Pengzhou Cheng, Gongshen Liu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.