Bangquan Xie
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Bangquan Xie has a total of 34 co-authors. Here are the top co-authors by number of shared publications:
Xiaoxiong Weng
School of Civil Engineering Transportation, South China University of Technology, Guangzhou, China
6 shared publications- Bing Li
Department of Automotive Engineering, Clemson University International Center for Automotive Research (CU-ICAR), Greenville, SC, USA
5 shared publications - Liang Yang
Apple, Inc
5 shared publications - Ailin Wei
Department of Bioengineering, Clemson University, Clemson, SC, USA
4 shared publications Ruifa Luo
Shenzhen Genvict Technologies co.Ltd. Shenzhen, China
3 shared publications- Shenghuang He
Dongguan University of Technology
3 shared publications
Latest Publications
- 1 December 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (12) , 13701-13708
FlippingAGR: A Flipping Aerial-Ground Robot With Dual-Mode Terrestrial Locomotion
- 1 December 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (12) , 13701-13708
FlippingAGR: A Flipping Aerial-Ground Robot With Dual-Mode Terrestrial Locomotion
- 25 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 23 (15455955) , 10186-10200
Reinforcement Learning Neural Network Observer-Based Adaptive Optimal Time-Varying Formation Control for Uncertain Multi-Agent Systems With External Disturbance
The traditional formation control methods of multi-agent systems (MASs) often rely on restrictive linear matrix inequalities and strong assumptions. To overcome this limitation, this work proposes an adaptive optimal time-varying formation control protocol for uncertain MASs, integrating reinforcement learning (RL)-based neural network observer. First, a radial basis function neural network (RBFNN) with adaptive law is designed to approximate the unknown nonlinear dynamics. An RBFNN-based state estimator and a novel disturbance estimator are developed to reconstruct the unmeasurable system state and the disturbance generated by an exosystem, respectively. Second, using the state estimate, a distributed formation control error system is established, and a performance index based on a Hamilton–Jacobi–Bellman (HJB) equation is introduced to achieve the objective of optimal formation control. Third, leveraging the state and disturbance estimates, an actor-critic RL-based optimal time-varying formation control protocol with adaptive update laws is proposed. Actor and critic networks are constructed to approximate unknown nonlinear terms in HJB equation. It is proved that estimator error systems and RBFNN approximation errors are semi-globally uniformly ultimately bounded (SGUUB). Simulation examples are provided to verify the effectiveness of the proposed estimators and optimal formation control protocol. Moreover, the method is successfully applied to unmanned aerial vehicles (UAVs) formation control in the Isaac Sim simulation environment, demonstrating the method’s practical feasibility.
- 25 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 23 (15455955) , 10186-10200
Reinforcement Learning Neural Network Observer-Based Adaptive Optimal Time-Varying Formation Control for Uncertain Multi-Agent Systems With External Disturbance
The traditional formation control methods of multi-agent systems (MASs) often rely on restrictive linear matrix inequalities and strong assumptions. To overcome this limitation, this work proposes an adaptive optimal time-varying formation control protocol for uncertain MASs, integrating reinforcement learning (RL)-based neural network observer. First, a radial basis function neural network (RBFNN) with adaptive law is designed to approximate the unknown nonlinear dynamics. An RBFNN-based state estimator and a novel disturbance estimator are developed to reconstruct the unmeasurable system state and the disturbance generated by an exosystem, respectively. Second, using the state estimate, a distributed formation control error system is established, and a performance index based on a Hamilton–Jacobi–Bellman (HJB) equation is introduced to achieve the objective of optimal formation control. Third, leveraging the state and disturbance estimates, an actor-critic RL-based optimal time-varying formation control protocol with adaptive update laws is proposed. Actor and critic networks are constructed to approximate unknown nonlinear terms in HJB equation. It is proved that estimator error systems and RBFNN approximation errors are semi-globally uniformly ultimately bounded (SGUUB). Simulation examples are provided to verify the effectiveness of the proposed estimators and optimal formation control protocol. Moreover, the method is successfully applied to unmanned aerial vehicles (UAVs) formation control in the Isaac Sim simulation environment, demonstrating the method’s practical feasibility.
- 29 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 56 (3) , 1721-1735
Enhanced Robustness in Simultaneous Fault Estimation and Distributed Fault-Tolerant Consensus Tracking Control for Multiagent Systems Utilizing Extended Observer Approach
In the context of consensus tracking control for multiagent systems (MASs), the system performance may be significantly degraded by multiple potential factors, including actuator/sensor faults and external disturbances. However, the development of a distributed fault estimation (FE) mechanism integrated with a fault-tolerant consensus control protocol remains a critical research challenge. This work proposes a novel distributed extended simultaneous observer (DESO) and observer-based fault-tolerant consensus tracking control (OB-FTCT) protocol for nonlinear MASs. First, system state as well as actuator and sensor faults for each agent are integrated into a new augmented vector. The original system is transformed into a descriptor system, and a DESO, tailored to the augmented vector, is designed to simultaneously estimate the system state and faults. Second, a novel OB-FTCT protocol is designed to achieve the H∞ consensus tracking and compensate the impact of the two faults. Through the establishment of Lyapunov functions and an algorithm, the stability of consensus tracking error system is proved and ensured. Third, the effectiveness of the proposed method is verified by simulation examples. Also, we apply the proposed method to single-link flexible joint robot, and compare it with existing method to prove the effectiveness.
- 29 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 56 (3) , 1721-1735
Enhanced Robustness in Simultaneous Fault Estimation and Distributed Fault-Tolerant Consensus Tracking Control for Multiagent Systems Utilizing Extended Observer Approach
In the context of consensus tracking control for multiagent systems (MASs), the system performance may be significantly degraded by multiple potential factors, including actuator/sensor faults and external disturbances. However, the development of a distributed fault estimation (FE) mechanism integrated with a fault-tolerant consensus control protocol remains a critical research challenge. This work proposes a novel distributed extended simultaneous observer (DESO) and observer-based fault-tolerant consensus tracking control (OB-FTCT) protocol for nonlinear MASs. First, system state as well as actuator and sensor faults for each agent are integrated into a new augmented vector. The original system is transformed into a descriptor system, and a DESO, tailored to the augmented vector, is designed to simultaneously estimate the system state and faults. Second, a novel OB-FTCT protocol is designed to achieve the H∞ consensus tracking and compensate the impact of the two faults. Through the establishment of Lyapunov functions and an algorithm, the stability of consensus tracking error system is proved and ensured. Third, the effectiveness of the proposed method is verified by simulation examples. Also, we apply the proposed method to single-link flexible joint robot, and compare it with existing method to prove the effectiveness.
- 27 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 71 (4) , 11588-11600
Distributed Deep Reinforcement Learning for Ackermann Multi-Robot Formation: A Weighted Multi-Objective Optimization
The integration of Internet of Things (IoT) with the multi-robot formation problem has been successfully applied in autonomous vehicles, unmanned aerial vehicles (UAVs), and industrial robotics, enhancing robot collaboration in complex tasks and increasing their autonomy when interacting with uncertain environments. However, extending formation strategies to Ackermann-steering robots remains challenging due to the non-holonomic motion constraints inherent in their kinematics. To address this, we propose the Ackermann Multi-Objective Deep Reinforcement Learning (AM-DRL) framework, a distributed deep reinforcement learning (DDRL) approach specifically tailored for Ackermann-type dynamics. First, we design a novel reward function that incorporates projection, distance, alignment, curvature, and deceleration terms, allowing more effective policy learning under real-world driving constraints. Second, we introduce a dynamic weighted parameter-sharing mechanism that prioritizes agent contributions during model updates based on individual performance, thus accelerating convergence and reducing the risk of local optima. Third, we formulate multi-objective policy optimization (MOPO) using distinct actor-critic networks to enhance policy differentiation across heterogeneous agents. Experimental results show that, compared to baseline algorithms without weighted parameter sharing, the proposed method reduces the average distance error by 16.65% and improves formation integrity by 35.14%, demonstrating superior convergence, trajectory smoothness, and robustness.
- 27 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 71 (4) , 11588-11600
Distributed Deep Reinforcement Learning for Ackermann Multi-Robot Formation: A Weighted Multi-Objective Optimization
The integration of Internet of Things (IoT) with the multi-robot formation problem has been successfully applied in autonomous vehicles, unmanned aerial vehicles (UAVs), and industrial robotics, enhancing robot collaboration in complex tasks and increasing their autonomy when interacting with uncertain environments. However, extending formation strategies to Ackermann-steering robots remains challenging due to the non-holonomic motion constraints inherent in their kinematics. To address this, we propose the Ackermann Multi-Objective Deep Reinforcement Learning (AM-DRL) framework, a distributed deep reinforcement learning (DDRL) approach specifically tailored for Ackermann-type dynamics. First, we design a novel reward function that incorporates projection, distance, alignment, curvature, and deceleration terms, allowing more effective policy learning under real-world driving constraints. Second, we introduce a dynamic weighted parameter-sharing mechanism that prioritizes agent contributions during model updates based on individual performance, thus accelerating convergence and reducing the risk of local optima. Third, we formulate multi-objective policy optimization (MOPO) using distinct actor-critic networks to enhance policy differentiation across heterogeneous agents. Experimental results show that, compared to baseline algorithms without weighted parameter sharing, the proposed method reduces the average distance error by 16.65% and improves formation integrity by 35.14%, demonstrating superior convergence, trajectory smoothness, and robustness.
- 17 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 25 (17) , 32127-32136
CLIP-Adaptive Visual Inspection Framework for Age Recognition of Pericarpium Citri Reticulatae
The accurate identification of the storage age of pericarpium citri reticulatae (PCR) is crucial for assessing its medicinal value. However, the existing approaches face two key limitations: 1) the feature representation ability of the backbone is often insufficient due to the limited data and domain gap and 2) full fine-tuning of the backbone and complex model designs lead to increased training costs and suboptimal performance. To address these issues, a CLIP -adaptive visual inspection framework is developed, consisting of two crucial components: a CLIP-based feature extraction module and an adaptive decision module. Specifically, the former utilizes a frozen CLIP backbone to extract high-level semantic representations. The latter addresses the high training costs and performance limitations of the model through a meticulously designed adaptive decision module. Extensive experiments on two PCR age recognition datasets demonstrate the effectiveness of the proposed approach, achieving 99.84% (98.67%) accuracy, 99.85% (98.67%) precision, 99.84% (98.67%) recall, and 99.84% (98.66%) ${F}1$ score, respectively. Furthermore, the proposed method reduces the number of trainable parameters by a factor of 21, significantly outperforming the state-of-the-art methods. The source code and models will be publicly available at https://github.com/JiaMingZhong2621/PCR
- 17 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 25 (17) , 32127-32136
CLIP-Adaptive Visual Inspection Framework for Age Recognition of Pericarpium Citri Reticulatae
The accurate identification of the storage age of pericarpium citri reticulatae (PCR) is crucial for assessing its medicinal value. However, the existing approaches face two key limitations: 1) the feature representation ability of the backbone is often insufficient due to the limited data and domain gap and 2) full fine-tuning of the backbone and complex model designs lead to increased training costs and suboptimal performance. To address these issues, a CLIP -adaptive visual inspection framework is developed, consisting of two crucial components: a CLIP-based feature extraction module and an adaptive decision module. Specifically, the former utilizes a frozen CLIP backbone to extract high-level semantic representations. The latter addresses the high training costs and performance limitations of the model through a meticulously designed adaptive decision module. Extensive experiments on two PCR age recognition datasets demonstrate the effectiveness of the proposed approach, achieving 99.84% (98.67%) accuracy, 99.85% (98.67%) precision, 99.84% (98.67%) recall, and 99.84% (98.66%) ${F}1$ score, respectively. Furthermore, the proposed method reduces the number of trainable parameters by a factor of 21, significantly outperforming the state-of-the-art methods. The source code and models will be publicly available at https://github.com/JiaMingZhong2621/PCR
- Open Access
- 20 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 25 (10) , 13277-13290
PedAST-GCN: Fast Pedestrian Crossing Intention Prediction Using Spatial–Temporal Attention Graph Convolution Networks
Accurately and timely predicting pedestrian crossing intentions in real-time is critical for operating intelligent vehicles on roads. Although existing models achieve promising accuracy using complex models and video image data, they are constrained for real-time practical use given the high model complexity, time-consuming data preprocessing, and low-quality image data in the wild. To address these, the paper proposes a Spatial-Temporal Attention Graph Convolution Network model for fast pedestrian crossing intention prediction (PedAST-GCN). It uses a lightweight GCN model as the backbone network with simple but robust graph representations of pedestrian crossing intention modality features, including pedestrian pose, bounding box, and vehicle speeds. The model is validated by comparing it with state-of-the-art models on two large-scale public datasets (JAAD and PIE). The results highlight the better performance of the PedAST-GCN model for pedestrian crossing intention prediction in terms of accuracy and computation times. The ablation analysis confirms the value of the backbone layer and graph design, the designed modality features, the effectiveness of attention mechanisms in capturing long-term dependencies (spatial-temporal attention) and fusing heterogeneous features (modality attention), and the robust performance across various observation lengths and in the presence of noisy data.
- 20 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 25 (10) , 13277-13290
Open AccessPedAST-GCN: Fast Pedestrian Crossing Intention Prediction Using Spatial–Temporal Attention Graph Convolution Networks
Accurately and timely predicting pedestrian crossing intentions in real-time is critical for operating intelligent vehicles on roads. Although existing models achieve promising accuracy using complex models and video image data, they are constrained for real-time practical use given the high model complexity, time-consuming data preprocessing, and low-quality image data in the wild. To address these, the paper proposes a Spatial-Temporal Attention Graph Convolution Network model for fast pedestrian crossing intention prediction (PedAST-GCN). It uses a lightweight GCN model as the backbone network with simple but robust graph representations of pedestrian crossing intention modality features, including pedestrian pose, bounding box, and vehicle speeds. The model is validated by comparing it with state-of-the-art models on two large-scale public datasets (JAAD and PIE). The results highlight the better performance of the PedAST-GCN model for pedestrian crossing intention prediction in terms of accuracy and computation times. The ablation analysis confirms the value of the backbone layer and graph design, the designed modality features, the effectiveness of attention mechanisms in capturing long-term dependencies (spatial-temporal attention) and fusing heterogeneous features (modality attention), and the robust performance across various observation lengths and in the presence of noisy data.
- Open Access
- 11 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 25 (2) , 2089-2100
SA-BiGCN: Bi-Stream Graph Convolution Networks With Spatial Attentions for the Eye Contact Detection in the Wild
Eye contact is essential in transmitting information and intention in the wild environment (e.g., urban streets or parking lots) with mixed vehicles and pedestrians. Compared with the vision image data, the human skeleton data are deemed to be robust to unconstrained surroundings and illumination. However, the skeleton graph-based approaches are mainly used for the action recognition. It is challenging to directly apply them to the eye detection task, which is momentary and dynamic given the complex wild environment. This paper proposes a Bi-stream Spatial Attention Graph Convolution Network (SA-BiGCN) for eye contact detection in the wild. We design a directed, nose-centric skeleton graph to capture relevant and hierarchical information and their interactions. We also propose a Bi-stream graph convolution network model with spatial attention to dynamically extract and fuse skeleton joints and bones information. The model was validated by comparing with state-of-art models on three large-scale public datasets, including JAAD, PIE, and LOOK. The results highlight the accuracy and generalization performance of the proposed SA-BiGCN model in detecting the eye contact in the wild environment. The ablation analysis validates the importance of the skeleton graph design, the spatial attention mechanism in the feature fusion process, as well as the model robustness against noisy skeleton data in terms of part occlusions, block occlusions, random occlusions, and random deviations.
- 11 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 25 (2) , 2089-2100
Open AccessSA-BiGCN: Bi-Stream Graph Convolution Networks With Spatial Attentions for the Eye Contact Detection in the Wild
Eye contact is essential in transmitting information and intention in the wild environment (e.g., urban streets or parking lots) with mixed vehicles and pedestrians. Compared with the vision image data, the human skeleton data are deemed to be robust to unconstrained surroundings and illumination. However, the skeleton graph-based approaches are mainly used for the action recognition. It is challenging to directly apply them to the eye detection task, which is momentary and dynamic given the complex wild environment. This paper proposes a Bi-stream Spatial Attention Graph Convolution Network (SA-BiGCN) for eye contact detection in the wild. We design a directed, nose-centric skeleton graph to capture relevant and hierarchical information and their interactions. We also propose a Bi-stream graph convolution network model with spatial attention to dynamically extract and fuse skeleton joints and bones information. The model was validated by comparing with state-of-art models on three large-scale public datasets, including JAAD, PIE, and LOOK. The results highlight the accuracy and generalization performance of the proposed SA-BiGCN model in detecting the eye contact in the wild environment. The ablation analysis validates the importance of the skeleton graph design, the spatial attention mechanism in the feature fusion process, as well as the model robustness against noisy skeleton data in terms of part occlusions, block occlusions, random occlusions, and random deviations.
- 1 August 2023
- Published by Elsevier in Pattern Recognition
- Vol. 144
ANAS: Asymptotic NAS for large-scale proxyless search and multi-task transfer learning
- 1 August 2023
- Published by Elsevier in Pattern Recognition
- Vol. 144
ANAS: Asymptotic NAS for large-scale proxyless search and multi-task transfer learning
- 1 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 4407-4415
MuTrans: Multiple Transformers for Fusing Feature Pyramid on 2D and 3D Object Detection
One of the major components of the neural network, the feature pyramid plays a vital part in perception tasks, like object detection in autonomous driving. But it is a challenge to fuse multi-level and multi-sensor feature pyramids for object detection. This paper proposes a simple yet effective framework named MuTrans ( Mu ltiple Trans formers) to fuse feature pyramid in single-stream 2D detector or two-stream 3D detector. The MuTrans based on encoder-decoder focuses on the significant features via multiple Transformers. MuTrans encoder uses three innovative self-attention mechanisms: S patial-wise B oxAlign attention ( SB ) for low-level spatial locations, C ontext-wise A ffinity attention ( CA ) for high-level context information, and high-level attention for multi-level features. Then MuTrans decoder processes these significant proposals including the RoI and context affinity. Besides, the L ow and H igh-level F usion ( LHF ) in the encoder reduces the number of computational parameters. And the Pre-LN [1] is utilized to accelerate the training convergence. LHF and Pre-LN are proven to reduce self-attention’s computational complexity and slow training convergence. Our result demonstrates the higher detection accuracy of MuTrans than that of the baseline method, particularly in small object detection. MuTrans demonstrates a 2.1 higher detection accuracy on APS index in small object detection on MS-COCO 2017 with ResNeXt-101 backbone, a 2.18 higher 3D detection accuracy (moderate difficulty) for small object-pedestrian on KITTI, and 6.85 higher RC index (Town05 Long) on CARLA urban driving simulator platform.
- 1 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 4407-4415
MuTrans: Multiple Transformers for Fusing Feature Pyramid on 2D and 3D Object Detection
One of the major components of the neural network, the feature pyramid plays a vital part in perception tasks, like object detection in autonomous driving. But it is a challenge to fuse multi-level and multi-sensor feature pyramids for object detection. This paper proposes a simple yet effective framework named MuTrans ( Mu ltiple Trans formers) to fuse feature pyramid in single-stream 2D detector or two-stream 3D detector. The MuTrans based on encoder-decoder focuses on the significant features via multiple Transformers. MuTrans encoder uses three innovative self-attention mechanisms: S patial-wise B oxAlign attention ( SB ) for low-level spatial locations, C ontext-wise A ffinity attention ( CA ) for high-level context information, and high-level attention for multi-level features. Then MuTrans decoder processes these significant proposals including the RoI and context affinity. Besides, the L ow and H igh-level F usion ( LHF ) in the encoder reduces the number of computational parameters. And the Pre-LN [1] is utilized to accelerate the training convergence. LHF and Pre-LN are proven to reduce self-attention’s computational complexity and slow training convergence. Our result demonstrates the higher detection accuracy of MuTrans than that of the baseline method, particularly in small object detection. MuTrans demonstrates a 2.1 higher detection accuracy on APS index in small object detection on MS-COCO 2017 with ResNeXt-101 backbone, a 2.18 higher 3D detection accuracy (moderate difficulty) for small object-pedestrian on KITTI, and 6.85 higher RC index (Town05 Long) on CARLA urban driving simulator platform.
- 16 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. PP (15249050) , 1-10
AMMF: Attention-Based Multi-Phase Multi-Task Fusion for Small Contour Object 3D Detection
Recently significant progress has been made in 3D detection. However, it is still challenging to detect small contour objects under complex scenes. This paper proposes a novel Attention-based Multi-phase Multi-task Fusion (AMMF) that uses point-level, RoI-level, and multi-task fusions to complement the disadvantages of LiDAR and camera, to solve this challenge. First, at the feature extraction phase, AMMF uses the Low and High-level Fusion with Matching Attention (LHF-MA) and efficient FPN (eFPN) to perform point-level fusion for cross sensors and single sensor, respectively. Instead of merging each level and using expensive 3D CNN like other methods, LHF-MA fuses low-level spatial location and high-level contextual feature of 2D CNN customized feature extractors and ignores the fusion of middle levels, reducing the computational cost. Then, at the proposal generation phase, Progressive Proposal Fusion (PPF) with learned attention map is used to perform coarse-to-fine RoI-level fusion, instead of only combining coarse-grained features at high-level of network. PPF using progressively increasing IoU thresholds could avoid overfitting and improve the performance. Note that the matching attentions and learned attention maps are utilized to weigh the priority of different sensors. Moreover, to solve the sparseness of point-wise fusion between LiDAR BEV and RGB image, AMMF uses multi-task fusion that generates pseudo-LiDAR from camera by depth estimation task, to guide this point-wise fusion. Finally, AMMF performs excellently for detecting small contour objects like pedestrians, cyclists, and distant cars. On the KITTI, AMMF finishes 3.62% improvements in the moderate instance for pedestrians. It achieves a 2.21% improvement in the $>$ 50 instance of LEVEL $_{-}$ 2 level for vehicle on the Waymo Open Dataset. And AMMF is further verified on our customized dataset consisting of challenging scenarios like strong illumination and heavy shadow cases.
- 16 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. PP (15249050) , 1-10
AMMF: Attention-Based Multi-Phase Multi-Task Fusion for Small Contour Object 3D Detection
Recently significant progress has been made in 3D detection. However, it is still challenging to detect small contour objects under complex scenes. This paper proposes a novel Attention-based Multi-phase Multi-task Fusion (AMMF) that uses point-level, RoI-level, and multi-task fusions to complement the disadvantages of LiDAR and camera, to solve this challenge. First, at the feature extraction phase, AMMF uses the Low and High-level Fusion with Matching Attention (LHF-MA) and efficient FPN (eFPN) to perform point-level fusion for cross sensors and single sensor, respectively. Instead of merging each level and using expensive 3D CNN like other methods, LHF-MA fuses low-level spatial location and high-level contextual feature of 2D CNN customized feature extractors and ignores the fusion of middle levels, reducing the computational cost. Then, at the proposal generation phase, Progressive Proposal Fusion (PPF) with learned attention map is used to perform coarse-to-fine RoI-level fusion, instead of only combining coarse-grained features at high-level of network. PPF using progressively increasing IoU thresholds could avoid overfitting and improve the performance. Note that the matching attentions and learned attention maps are utilized to weigh the priority of different sensors. Moreover, to solve the sparseness of point-wise fusion between LiDAR BEV and RGB image, AMMF uses multi-task fusion that generates pseudo-LiDAR from camera by depth estimation task, to guide this point-wise fusion. Finally, AMMF performs excellently for detecting small contour objects like pedestrians, cyclists, and distant cars. On the KITTI, AMMF finishes 3.62% improvements in the moderate instance for pedestrians. It achieves a 2.21% improvement in the $>$ 50 instance of LEVEL $_{-}$ 2 level for vehicle on the Waymo Open Dataset. And AMMF is further verified on our customized dataset consisting of challenging scenarios like strong illumination and heavy shadow cases.