Haiyun Guo
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Haiyun Guo has a total of 65 co-authors. Here are the top co-authors by number of shared publications:
- Jinqiao Wang
Chinese Academy of Sciences
18 shared publications - Ming Tang
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
11 shared publications - Kuan Zhu
Chinese Academy of Sciences, Foundation Model Research Center, Institute of Automation, Beijing, China
8 shared publications Jinqiao Wang
4 shared publicationsChaoyang Zhao
Institute of Automation National Laboratory of Pattern Recognition CAS Beijing China
3 shared publications- Min Huang
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
3 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 2912-2921
PLUME: Latent Reasoning Based Universal Multimodal Embedding
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 2912-2921
Open AccessPLUME: Latent Reasoning Based Universal Multimodal Embedding
- Open Access
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 1157-1165
HB-Mamba: Hierarchical Bi-directional State Space Modeling for LiDAR Semantic Segmentation in Autonomous Driving
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 1157-1165
Open AccessHB-Mamba: Hierarchical Bi-directional State Space Modeling for LiDAR Semantic Segmentation in Autonomous Driving
- Top Cited Papers
- 12 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 2699-2713
Continual Instruction Tuning for Large Multimodal Models
Instruction tuning has become a widely adopted approach for aligning large multimodal models (LMMs) with human intent. It enables multi-task joint training through unified data formats. However, as new vision-language tasks constantly emerge, exhaustive joint training of all tasks becomes impractical. Continual learning offers a more flexible and resource-efficient alternative, enabling incremental training of LMMs on emerging tasks. This study investigates two fundamental questions when applying continual learning to instruction tuning of LMMs: 1) Do LMMs suffer from catastrophic forgetting during continual instruction tuning? 2) Can existing continual learning methods be effectively applied to continual instruction tuning of LMMs? A comprehensive study was conducted to answer these questions. First, we establish the first benchmark for continual instruction tuning of LMMs and reveal the phenomenon of catastrophic forgetting in this setup. Second, we integrate and adapt traditional continual learning approaches to this setting, demonstrating the effectiveness of these strategies to varying degrees in different scenarios. Third, we explore task-similarity dynamics between pairs of vision-language tasks and propose task-similarity-informed regularization and model expansion methods. Experimental results show that our approach can consistently boost the model’s performance.
- 12 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 2699-2713
Top Cited PapersContinual Instruction Tuning for Large Multimodal Models
Instruction tuning has become a widely adopted approach for aligning large multimodal models (LMMs) with human intent. It enables multi-task joint training through unified data formats. However, as new vision-language tasks constantly emerge, exhaustive joint training of all tasks becomes impractical. Continual learning offers a more flexible and resource-efficient alternative, enabling incremental training of LMMs on emerging tasks. This study investigates two fundamental questions when applying continual learning to instruction tuning of LMMs: 1) Do LMMs suffer from catastrophic forgetting during continual instruction tuning? 2) Can existing continual learning methods be effectively applied to continual instruction tuning of LMMs? A comprehensive study was conducted to answer these questions. First, we establish the first benchmark for continual instruction tuning of LMMs and reveal the phenomenon of catastrophic forgetting in this setup. Second, we integrate and adapt traditional continual learning approaches to this setting, demonstrating the effectiveness of these strategies to varying degrees in different scenarios. Third, we explore task-similarity dynamics between pairs of vision-language tasks and propose task-similarity-informed regularization and model expansion methods. Experimental results show that our approach can consistently boost the model’s performance.
- Open Access
R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training
Self-play bootstraps LLM reasoning through an iterative Challenger-Solver loop: the Challenger is trained to generate questions that target the Solver's capabilities, and the Solver is optimized on the generated data to expand its reasoning skills. However, existing frameworks like R-Zero often exhibit non-sustained improvement, where early gains degrade as self-play continues. We identify a key failure mode, Diversity Illusion, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns. It manifests as (1) Local Diversity Illusion, where diversity is enforced only within-batch, inducing cross-iteration mode cycling; and (2) Surface Diversity Illusion, where questions vary superficially but require near-identical reasoning skills. To mitigate them, we propose R-Diverse with two aligned innovations: Memory-Augmented Penalty (MAP), which uses a persistent memory bank to discourage recycling across iterations, and Skill-Aware Measurement (SAM), which evaluates diversity by the reasoning skills exercised rather than surface variation of questions. Across 10 math and general reasoning benchmarks, R-Diverse sustains gains over more iterations and consistently outperforms prior self-play methods. Code is available at https://github.com/Gengsheng-Li/R-Diverse.
Open AccessR-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training
Self-play bootstraps LLM reasoning through an iterative Challenger-Solver loop: the Challenger is trained to generate questions that target the Solver's capabilities, and the Solver is optimized on the generated data to expand its reasoning skills. However, existing frameworks like R-Zero often exhibit non-sustained improvement, where early gains degrade as self-play continues. We identify a key failure mode, Diversity Illusion, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns. It manifests as (1) Local Diversity Illusion, where diversity is enforced only within-batch, inducing cross-iteration mode cycling; and (2) Surface Diversity Illusion, where questions vary superficially but require near-identical reasoning skills. To mitigate them, we propose R-Diverse with two aligned innovations: Memory-Augmented Penalty (MAP), which uses a persistent memory bank to discourage recycling across iterations, and Skill-Aware Measurement (SAM), which evaluates diversity by the reasoning skills exercised rather than surface variation of questions. Across 10 math and general reasoning benchmarks, R-Diverse sustains gains over more iterations and consistently outperforms prior self-play methods. Code is available at https://github.com/Gengsheng-Li/R-Diverse.
- 30 January 2026
- Published by Springer Nature
- p. 136-150
MaSA: Mamba-Based Global Feature Selective Aggregator for Efficient Lane Detection
- 30 January 2026
- Published by Springer Nature
- p. 136-150
MaSA: Mamba-Based Global Feature Selective Aggregator for Efficient Lane Detection
- Open Access
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input-expert specialization. We term this phenomenon Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting.To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE-LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE-LoRA variants in both accuracy and anti-forgetting without adding parameters. Our code will be released upon acceptance.
Open AccessPASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input-expert specialization. We term this phenomenon Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting.To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE-LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE-LoRA variants in both accuracy and anti-forgetting without adding parameters. Our code will be released upon acceptance.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 4464-4473
Referring Expression Instance Retrieval and A Strong End-to-End Baseline
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 4464-4473
Open AccessReferring Expression Instance Retrieval and A Strong End-to-End Baseline
- 20 March 2024
- Published by Springer Nature in Machine Intelligence Research
- Vol. 21 (3) , 514-525
Structural Dependence Learning Based on Self-attention for Face Alignment
- 20 March 2024
- Published by Springer Nature in Machine Intelligence Research
- Vol. 21 (3) , 514-525
Structural Dependence Learning Based on Self-attention for Face Alignment
- 28 December 2023
- Published by Springer Nature
- p. 238-250
Instance-Proxy Loss for Semi-supervised Learning with Coarse Labels
- 28 December 2023
- Published by Springer Nature
- p. 238-250
Instance-Proxy Loss for Semi-supervised Learning with Coarse Labels
- Top Cited Papers
- 25 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (12) , 17307-17317
AAformer: Auto-Aligned Transformer for Person Re-Identification
In person re-identification (re-ID), extracting part-level features from person images has been verified to be crucial to offer fine-grained information. Most of the existing CNN-based methods only locate the human parts coarsely, or rely on pretrained human parsing models and fail in locating the identifiable nonhuman parts (e.g., knapsack). In this article, we introduce an alignment scheme in transformer architecture for the first time and propose the auto-aligned transformer (AAformer) to automatically locate both the human parts and nonhuman ones at patch level. We introduce the “Part tokens (PARTs)”, which are learnable vectors, to extract part features in the transformer. A PART only interacts with a local subset of patches in self-attention and learns to be the part representation. To adaptively group the image patches into different subsets, we design the auto-alignment. Auto-alignment employs a fast variant of optimal transport (OT) algorithm to online cluster the patch embeddings into several groups with the PARTs as their prototypes. AAformer integrates the part alignment into the self-attention and the output PARTs can be directly used as part features for retrieval. Extensive experiments validate the effectiveness of PARTs and the superiority of AAformer over various state-of-the-art methods.
- 25 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (12) , 17307-17317
Top Cited PapersAAformer: Auto-Aligned Transformer for Person Re-Identification
In person re-identification (re-ID), extracting part-level features from person images has been verified to be crucial to offer fine-grained information. Most of the existing CNN-based methods only locate the human parts coarsely, or rely on pretrained human parsing models and fail in locating the identifiable nonhuman parts (e.g., knapsack). In this article, we introduce an alignment scheme in transformer architecture for the first time and propose the auto-aligned transformer (AAformer) to automatically locate both the human parts and nonhuman ones at patch level. We introduce the “Part tokens (PARTs)”, which are learnable vectors, to extract part features in the transformer. A PART only interacts with a local subset of patches in self-attention and learns to be the part representation. To adaptively group the image patches into different subsets, we design the auto-alignment. Auto-alignment employs a fast variant of optimal transport (OT) algorithm to online cluster the patch embeddings into several groups with the PARTs as their prototypes. AAformer integrates the part alignment into the self-attention and the output PARTs can be directly used as part features for retrieval. Extensive experiments validate the effectiveness of PARTs and the superiority of AAformer over various state-of-the-art methods.