Wenjing Hou
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Wenjing Hou has a total of 51 co-authors. Here are the top co-authors by number of shared publications:
- Hong Wen
School of Aeronautics and Astronautics, University of Electronic Science and Technology of China, Chengdu 611731, China
16 shared publications - Wenxin Lei
School of Aeronautics and Astronautics, University of Electronic Science and Technology of China, Chengdu, China
6 shared publications - Jinsong Wu
Universidad de Chile, Santiago, Chile
4 shared publications - Zhiwei Chen
School of Aeronautics and Astronautics, University of Electronic Science and Technology of China, Chengdu, China
4 shared publications - Runhui Zhao
School of Aeronautics and Astronautics, University of Electronic Science and Technology of China, Chengdu, China
3 shared publications - Zhibo Pang
Peking University
3 shared publications
Latest Publications
- Open Access
Edge-Based Intelligent Task Management for Mobile Airfield Lighting Control
Airfield lighting control (ALC) is critical for ensuring safe, efficient, and compliant airport operations, especially under low-visibility conditions. However, current centralized control architectures cannot adequately meet the real-time responsiveness, scalability, and reliability requirements of Advanced Surface Movement Guidance and Control Systems (A-SMGCS) Level IV. To overcome these limitations, this paper proposes a novel cloud–edge–end collaborative architecture for a mobile ALC scenario, in which we formulate a joint task computing and energy consumption optimization problem to maximize long-term system utility under latency, computation, and communication constraints. In this way, the mobile airfield lighting (MAL) system can also quickly adapt its optimal formation pattern based on the airport environment, lighting conditions, and the type of aircraft taking off or landing via efficient computation, thereby achieving the best navigational assistance effect. For solving such an optimization problem, a framework that combines K-medoids with the Improved Twin Delayed Deep Deterministic Policy Gradient (ITD3) is proposed to integrate the efficiency of clustering for rough allocation and the high-precision dynamic optimization capability of the improved TD3. The training depends on edge nodes and the cloud to achieve online performance. Finally, the extensive simulation proved that our novel algorithm is efficient.
Open AccessEdge-Based Intelligent Task Management for Mobile Airfield Lighting Control
Airfield lighting control (ALC) is critical for ensuring safe, efficient, and compliant airport operations, especially under low-visibility conditions. However, current centralized control architectures cannot adequately meet the real-time responsiveness, scalability, and reliability requirements of Advanced Surface Movement Guidance and Control Systems (A-SMGCS) Level IV. To overcome these limitations, this paper proposes a novel cloud–edge–end collaborative architecture for a mobile ALC scenario, in which we formulate a joint task computing and energy consumption optimization problem to maximize long-term system utility under latency, computation, and communication constraints. In this way, the mobile airfield lighting (MAL) system can also quickly adapt its optimal formation pattern based on the airport environment, lighting conditions, and the type of aircraft taking off or landing via efficient computation, thereby achieving the best navigational assistance effect. For solving such an optimization problem, a framework that combines K-medoids with the Improved Twin Delayed Deep Deterministic Policy Gradient (ITD3) is proposed to integrate the efficiency of clustering for rough allocation and the high-precision dynamic optimization capability of the improved TD3. The training depends on edge nodes and the cloud to achieve online performance. Finally, the extensive simulation proved that our novel algorithm is efficient.
- Open Access
Digital Twin-Driven TLE Error Correction for Precise LEO Satellite Orbit Prediction
Low earth orbit (LEO) satellite orbit prediction is one of the key measures to compensate for position errors and ensure position accuracy, which plays an important role in the aerospace communication network for undertaking functions such as routing relay, real-time communication, and signal forwarding. However, existing learning-based satellite orbit prediction models that are recognized as the best measurement inevitably face the problem of distribution bias. Orbit predictions can lead to a decrease in model performance due to different types of satellites (LEO and SSO) and different time scales. In this article, a new method is explored to overcome these shortcomings. Unlike previous methods that explore the temporal correlation of orbit data, this novel orbit prediction method converts satellite orbit data into the frequency domain via Fourier transformation, using a third-order Fourier-derivative convolution framework. Specifically, the proposed Fourier dilation convolution (FDC) model demonstrates better generalization ability across different types of satellites and different time scales by combining frequency domain analysis and dilated convolution. Two real datasets are applied for experimental validation, and the results show the effectiveness of our proposed FDC model. Meanwhile, the proposed FDC model shows a decrease in mean absolute error (MAE) values compared to the temporal convolutional network based seasonal and trend decomposition using a Loess (STL-TCN) model. Quantitative comparisons demonstrate that compared to the STL-TCN model, the FDC model reduces the mean absolute error (MAE) by approximately 10% to 85% across different orbital dimensions. Finally, we conducted further analysis of the interpretability of the model.
Open AccessDigital Twin-Driven TLE Error Correction for Precise LEO Satellite Orbit Prediction
Low earth orbit (LEO) satellite orbit prediction is one of the key measures to compensate for position errors and ensure position accuracy, which plays an important role in the aerospace communication network for undertaking functions such as routing relay, real-time communication, and signal forwarding. However, existing learning-based satellite orbit prediction models that are recognized as the best measurement inevitably face the problem of distribution bias. Orbit predictions can lead to a decrease in model performance due to different types of satellites (LEO and SSO) and different time scales. In this article, a new method is explored to overcome these shortcomings. Unlike previous methods that explore the temporal correlation of orbit data, this novel orbit prediction method converts satellite orbit data into the frequency domain via Fourier transformation, using a third-order Fourier-derivative convolution framework. Specifically, the proposed Fourier dilation convolution (FDC) model demonstrates better generalization ability across different types of satellites and different time scales by combining frequency domain analysis and dilated convolution. Two real datasets are applied for experimental validation, and the results show the effectiveness of our proposed FDC model. Meanwhile, the proposed FDC model shows a decrease in mean absolute error (MAE) values compared to the temporal convolutional network based seasonal and trend decomposition using a Loess (STL-TCN) model. Quantitative comparisons demonstrate that compared to the STL-TCN model, the FDC model reduces the mean absolute error (MAE) by approximately 10% to 85% across different orbital dimensions. Finally, we conducted further analysis of the interpretability of the model.
- 12 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. PP (00983063) , 1
Anti-Jamming Communication Framework for UAV based Defense Operations to Control AI-enabled Attacks
Nowadays, unmanned aerial vehicles (UAVs) have become essential assets for defense operations, providing efficient and precise communication and surveillance capabilities in complex and hostile environments. Artificial intelligence (AI)-enabled jamming poses a critical threat to the security and reliability of UAV communication networks. To address this, we propose an offline-learning anti-jamming framework that leverages advanced algorithms, including deep reinforcement learning (DRL) and decision transformers (DT), to enhance UAV performance in intelligent jamming environments. This work adopts an offline, cloud-based training paradigm for anti-jamming, which learns from historical decision trajectories to handle uncertain and unseen jamming patterns and enable reliable decision making. The proposed framework constrains cumulative reward loss within a predefined bound while improving training stability and reducing the time complexity of DRL-based decision model training. The proposed framework exhibits greater performance of UAVs in controlling AI-enabled jams by showing remarkable results for the performance metrics such as Bit Error Rate (BER), Signal-to-Noise Ratio (SNR), Packet Delivery Ratio (PDR), Jamming Detection Time (JDT), and latency (delay in response). The extensive experimental outcomes illustrate the efficiency of the proposed framework in complex defense operations and uncertain environments for mitigating AI-enabled attacks by counteracting the jamming attempts.
- 12 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. PP (00983063) , 1
Anti-Jamming Communication Framework for UAV based Defense Operations to Control AI-enabled Attacks
Nowadays, unmanned aerial vehicles (UAVs) have become essential assets for defense operations, providing efficient and precise communication and surveillance capabilities in complex and hostile environments. Artificial intelligence (AI)-enabled jamming poses a critical threat to the security and reliability of UAV communication networks. To address this, we propose an offline-learning anti-jamming framework that leverages advanced algorithms, including deep reinforcement learning (DRL) and decision transformers (DT), to enhance UAV performance in intelligent jamming environments. This work adopts an offline, cloud-based training paradigm for anti-jamming, which learns from historical decision trajectories to handle uncertain and unseen jamming patterns and enable reliable decision making. The proposed framework constrains cumulative reward loss within a predefined bound while improving training stability and reducing the time complexity of DRL-based decision model training. The proposed framework exhibits greater performance of UAVs in controlling AI-enabled jams by showing remarkable results for the performance metrics such as Bit Error Rate (BER), Signal-to-Noise Ratio (SNR), Packet Delivery Ratio (PDR), Jamming Detection Time (JDT), and latency (delay in response). The extensive experimental outcomes illustrate the efficiency of the proposed framework in complex defense operations and uncertain environments for mitigating AI-enabled attacks by counteracting the jamming attempts.
- 9 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 12 (23) , 49248-49259
An Offline Learning Framework for Edge Resource Allocation and Task Offloading in Agricultural IoT With Limited Trajectory Data
In Agricultural Internet of Things (Ag-IoT), a large number of terminal sensor nodes with limited computational power and energy are widely deployed across various production sites to collect real-time data that support precision agriculture decision-making, which makes it challenging to perform complex computing tasks in real time. This article proposes an offline decision-making framework based on a compositional diffusion model that addresses the limitations of traditional offline deep reinforcement learning (DRL), which often struggles to produce high-quality policies in the presence of malicious-node interference and attacks and with insufficient decision trajectories in agricultural settings. Specifically, the proposed scheme leverages the diffusion model’s reverse generative process to synthesize numerous high-quality augmented samples from a small set of existing decision trajectories, which are then used to train robust policies, mitigating performance degradation in offline DRL due to limited trajectories and adversarial state perturbations introduced by malicious nodes. Experimental results demonstrate that the proposed method improves the quality and generalization of task offloading and resource allocation decisions, exhibiting superior stability and performance even under data sparsity and adversarial attacks.
- 9 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 12 (23) , 49248-49259
An Offline Learning Framework for Edge Resource Allocation and Task Offloading in Agricultural IoT With Limited Trajectory Data
In Agricultural Internet of Things (Ag-IoT), a large number of terminal sensor nodes with limited computational power and energy are widely deployed across various production sites to collect real-time data that support precision agriculture decision-making, which makes it challenging to perform complex computing tasks in real time. This article proposes an offline decision-making framework based on a compositional diffusion model that addresses the limitations of traditional offline deep reinforcement learning (DRL), which often struggles to produce high-quality policies in the presence of malicious-node interference and attacks and with insufficient decision trajectories in agricultural settings. Specifically, the proposed scheme leverages the diffusion model’s reverse generative process to synthesize numerous high-quality augmented samples from a small set of existing decision trajectories, which are then used to train robust policies, mitigating performance degradation in offline DRL due to limited trajectories and adversarial state perturbations introduced by malicious nodes. Experimental results demonstrate that the proposed method improves the quality and generalization of task offloading and resource allocation decisions, exhibiting superior stability and performance even under data sparsity and adversarial attacks.
- 20 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Network
- Vol. 39 (4) , 29-36
Wireless Security Enhancement Framework Based on AI and RIS in Industrial IoT Networks
Reconfigurable Intelligent Surface (RIS) technology, closely integrated with Artificial Intelligence (AI) as one of the Physical Layer (PL) technology approaches, is considered an ef-fective solution for enhancing security in industrial Internet of Things (IIoT) networks. Integrating RIS, AI, and heterogeneous IIoT networks presents a promising approach to improving the se-curity of wireless communications in IIoT. This paper discusses new AI-based Physical Layer Authentication (PLA) approaches and how they can be integrated with RIS to enable identity authen-tication and malicious node detection, aiming to provide light-weight isolation in IIoT terminals. An anti-jamming and secure communication framework for IIoT is proposed, which leverages Deep Reinforcement Learning (DRL) and RIS to defend against malicious jamming and eavesdropping targeting legitimate users' channels. Additionally, a foundational framework that employs Quantum Reinforcement Learning (QRL) is presented to improve RIS-assisted secure communication. Furthermore, the threats posed by Generative AI (GAI) to RIS-assisted IIoT communica-tion networks and possible defense schemes are provided. Finally, the paper concludes by discussing research opportunities and fu-ture challenges.
- 20 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Network
- Vol. 39 (4) , 29-36
Wireless Security Enhancement Framework Based on AI and RIS in Industrial IoT Networks
Reconfigurable Intelligent Surface (RIS) technology, closely integrated with Artificial Intelligence (AI) as one of the Physical Layer (PL) technology approaches, is considered an ef-fective solution for enhancing security in industrial Internet of Things (IIoT) networks. Integrating RIS, AI, and heterogeneous IIoT networks presents a promising approach to improving the se-curity of wireless communications in IIoT. This paper discusses new AI-based Physical Layer Authentication (PLA) approaches and how they can be integrated with RIS to enable identity authen-tication and malicious node detection, aiming to provide light-weight isolation in IIoT terminals. An anti-jamming and secure communication framework for IIoT is proposed, which leverages Deep Reinforcement Learning (DRL) and RIS to defend against malicious jamming and eavesdropping targeting legitimate users' channels. Additionally, a foundational framework that employs Quantum Reinforcement Learning (QRL) is presented to improve RIS-assisted secure communication. Furthermore, the threats posed by Generative AI (GAI) to RIS-assisted IIoT communica-tion networks and possible defense schemes are provided. Finally, the paper concludes by discussing research opportunities and fu-ture challenges.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4246-4255
Causal Visual-semantic Correlation for Zero-shot Learning
Zero-Shot learning (ZSL) correlates visual samples and shared semantic information to transfer knowledge from seen classes to unseen classes. Existing methods typically establish visual-semantic correlation by aligning visual and semantic features, which are extracted from visual samples and semantic information, respectively. However, instance-level images, owing to singular observation perspectives and diverse individuals, cannot exactly match the comprehensive semantic information defined at the class level. Direct feature alignment imposes correlation between mismatched vision and semantics, resulting in spurious visual-semantic correlation. To address this, we propose a novel method termed Causal Visual-semantic Correlation (CVsC) to learn substantive visual-semantic correlation for ZSL. Specifically, we utilize a Visual Semantic Attention module to facilitate interaction between vision and semantics, thereby identifying attribute-related visual features. Furthermore, we design a Conditional Correlation Loss to properly utilize semantic information as supervision for establishing visual-semantic correlation. Moreover, we introduce counterfactual intervention applied to attribute-related visual features, and maximize their impact on semantic and target predictions to enhance substantive visual-semantic correlation. Extensive experiments conducted on three benchmark datasets (i.e., CUB, SUN, and AWA2) demonstrate that our CVSC outperforms existing state-of-the-art methods.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4246-4255
Open AccessCausal Visual-semantic Correlation for Zero-shot Learning
Zero-Shot learning (ZSL) correlates visual samples and shared semantic information to transfer knowledge from seen classes to unseen classes. Existing methods typically establish visual-semantic correlation by aligning visual and semantic features, which are extracted from visual samples and semantic information, respectively. However, instance-level images, owing to singular observation perspectives and diverse individuals, cannot exactly match the comprehensive semantic information defined at the class level. Direct feature alignment imposes correlation between mismatched vision and semantics, resulting in spurious visual-semantic correlation. To address this, we propose a novel method termed Causal Visual-semantic Correlation (CVsC) to learn substantive visual-semantic correlation for ZSL. Specifically, we utilize a Visual Semantic Attention module to facilitate interaction between vision and semantics, thereby identifying attribute-related visual features. Furthermore, we design a Conditional Correlation Loss to properly utilize semantic information as supervision for establishing visual-semantic correlation. Moreover, we introduce counterfactual intervention applied to attribute-related visual features, and maximize their impact on semantic and target predictions to enhance substantive visual-semantic correlation. Extensive experiments conducted on three benchmark datasets (i.e., CUB, SUN, and AWA2) demonstrate that our CVSC outperforms existing state-of-the-art methods.
- 24 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 70 (1) , 2391-2400
Cross-Device Radio Frequency Fingerprinting Identification Based on Domain Adaptation
Radio frequency fingerprinting (RFF) is a lightweight authentication technology for resource-limited terminal nodes by exploiting the unique hardware imperfections resulting from the manufacturing process. Previous studies about radio frequency fingerprinting identification (RFFI) mainly concentrate on improving the accuracy which is evaluated by the single receiver device that trains and identifies all the nodes. Due to the mobility of the consumer electronic terminals, these terminal nodes may need to be identified by the different receivers. In this paper, we propose a cross-device radio frequency fingerprinting identification scheme which allows enrolled nodes to be authenticated by different devices. Motivated by the observation that signals collected by different receiver devices have a distribution shift that would violate the basic independent and identically distributed (i.i.d) assumption of supervised learning. Domain adaptation is adopted to improve the accuracy under different receivers, which can align the data captured from different devices and eliminate the distribution shift through the labeled data from one receiver device and unlabeled data from the other device. By this way, the distribution shift from different devices is corrected. Extensive experiment configurations under various Signal-to-noise ratio (SNR) are carried out to demonstrate the performance of domain adaptation with the same model structure. The results indicate that classification accuracy under different devices can be increased by 7%-15% and get a stable accuracy rate higher than 90% by leveraging our proposed cross-device RFFI scheme.
- 24 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 70 (1) , 2391-2400
Cross-Device Radio Frequency Fingerprinting Identification Based on Domain Adaptation
Radio frequency fingerprinting (RFF) is a lightweight authentication technology for resource-limited terminal nodes by exploiting the unique hardware imperfections resulting from the manufacturing process. Previous studies about radio frequency fingerprinting identification (RFFI) mainly concentrate on improving the accuracy which is evaluated by the single receiver device that trains and identifies all the nodes. Due to the mobility of the consumer electronic terminals, these terminal nodes may need to be identified by the different receivers. In this paper, we propose a cross-device radio frequency fingerprinting identification scheme which allows enrolled nodes to be authenticated by different devices. Motivated by the observation that signals collected by different receiver devices have a distribution shift that would violate the basic independent and identically distributed (i.i.d) assumption of supervised learning. Domain adaptation is adopted to improve the accuracy under different receivers, which can align the data captured from different devices and eliminate the distribution shift through the labeled data from one receiver device and unlabeled data from the other device. By this way, the distribution shift from different devices is corrected. Extensive experiment configurations under various Signal-to-noise ratio (SNR) are carried out to demonstrate the performance of domain adaptation with the same model structure. The results indicate that classification accuracy under different devices can be increased by 7%-15% and get a stable accuracy rate higher than 90% by leveraging our proposed cross-device RFFI scheme.
- 18 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (3) , 4327-4336
Physical Layer Enhanced Zero-Trust Security for Wireless Industrial Internet of Things
As security issues facing the industrial Internet of Things (IIoT) continue to emerge, industrial organizations are working to further improve the security system. Zero trust (ZT) is seen as the future of industrial security, with a rising voice, but currently, no concrete implementation technique is available. In this article, we start with the requirements of ZT security and attempt to design a ZT technical framework applicable to wireless IIoT. Specifically, a three-step ZT security framework is proposed that builds on the benefits of physical-layer security to enhance ZT in IIoT. Security zone formation is done first, which then facilitates a trusted environment for subsequent device authentication and cryptographic negotiation. By integrating physical-layer security, several promising techniques, including artificial noise, physical fingerprint, and key distribution, are well designed to accomplish the proposed framework. Our analysis reveals that the proposed framework and the designed particular implementation techniques are feasible to enhance ZT security in wireless IIoT.
- 18 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (3) , 4327-4336
Physical Layer Enhanced Zero-Trust Security for Wireless Industrial Internet of Things
As security issues facing the industrial Internet of Things (IIoT) continue to emerge, industrial organizations are working to further improve the security system. Zero trust (ZT) is seen as the future of industrial security, with a rising voice, but currently, no concrete implementation technique is available. In this article, we start with the requirements of ZT security and attempt to design a ZT technical framework applicable to wireless IIoT. Specifically, a three-step ZT security framework is proposed that builds on the benefits of physical-layer security to enhance ZT in IIoT. Security zone formation is done first, which then facilitates a trusted environment for subsequent device authentication and cryptographic negotiation. By integrating physical-layer security, several promising techniques, including artificial noise, physical fingerprint, and key distribution, are well designed to accomplish the proposed framework. Our analysis reveals that the proposed framework and the designed particular implementation techniques are feasible to enhance ZT security in wireless IIoT.
- 27 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (5) , 4376-4388
Adaptive Training and Aggregation for Federated Learning in Multi-Tier Computing Networks
Multi-tier computing (MC) utilizes computing resources from the cloud, fog, edge, and end layers to promote intelligent Internet of Things (IoT) applications. Federated learning (FL) in MC offers a prospective distributed and privacy-preserving framework to deploy deep learning applications in different layers. Due to time-varying network topologies, wireless channel states, and computational workloads, MC faces a dynamic and uncertain environment, which poses additional challenges to FL task processing. In this paper, we propose a novel adaptive training and aggregation federated learning (ATAFL) framework. Specifically, local model training can be performed at end devices, edge nodes, and fog nodes. The global aggregator can choose from the edge, fog, and cloud layers. A joint optimization problem of training, aggregation node selection, and resource allocation is further formulated to minimize system latency and energy consumption. Moreover, digital twin and deep reinforcement learning (DRL) techniques are integrated into the MC network to design optimal node selection and resource allocation strategies based on captured state information of the MC system. We implement a prototype, and experimental results show that our proposed DRL-based algorithm reduces system latency and energy consumption compared with the benchmark algorithms.
- 27 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (5) , 4376-4388
Adaptive Training and Aggregation for Federated Learning in Multi-Tier Computing Networks
Multi-tier computing (MC) utilizes computing resources from the cloud, fog, edge, and end layers to promote intelligent Internet of Things (IoT) applications. Federated learning (FL) in MC offers a prospective distributed and privacy-preserving framework to deploy deep learning applications in different layers. Due to time-varying network topologies, wireless channel states, and computational workloads, MC faces a dynamic and uncertain environment, which poses additional challenges to FL task processing. In this paper, we propose a novel adaptive training and aggregation federated learning (ATAFL) framework. Specifically, local model training can be performed at end devices, edge nodes, and fog nodes. The global aggregator can choose from the edge, fog, and cloud layers. A joint optimization problem of training, aggregation node selection, and resource allocation is further formulated to minimize system latency and energy consumption. Moreover, digital twin and deep reinforcement learning (DRL) techniques are integrated into the MC network to design optimal node selection and resource allocation strategies based on captured state information of the MC system. We implement a prototype, and experimental results show that our proposed DRL-based algorithm reduces system latency and energy consumption compared with the benchmark algorithms.
- 15 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-17
TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning
Zero-shot learning (ZSL) tackles the novel class recognition problem by transferring semantic knowledge from seen classes to unseen ones. Semantic knowledge is typically represented by attribute descriptions shared between different classes, which act as strong priors for localizing object attributes that represent discriminative region features, enabling significant and sufficient visual-semantic interaction for advancing ZSL. Existing attention-based models have struggled to learn inferior region features in a single image by solely using unidirectional attention, which ignore the transferable and discriminative attribute localization of visual features for representing the key semantic knowledge for effective knowledge transfer in ZSL. In this paper, we propose a cross attribute-guided Transformer network, termed TransZero++, to refine visual features and learn accurate attribute localization for key semantic knowledge representations in ZSL. Specifically, TransZero++ employs an attribute $\rightarrow$ visual Transformer sub-net (AVT) and a visual $\rightarrow$ attribute Transformer sub-net (VAT) to learn attribute-based visual features and visual-based attribute features, respectively. By further introducing feature-level and prediction-level semantical collaborative losses, the two attribute-guided transformers teach each other to learn semantic-augmented visual embeddings for key semantic knowledge representations via semantical collaborative learning. Finally, the semantic-augmented visual embeddings learned by AVT and VAT are fused to conduct desirable visual-semantic interaction cooperated with class semantic vectors for ZSL classification. Extensive experiments show that TransZero++ achieves the new state-of-the-art results on three golden ZSL benchmarks and on the large-scale ImageNet dataset. The project website is available at: https://shiming-chen.github.io/TransZero-pp/TransZero-pp.html .
- 15 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-17
TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning
Zero-shot learning (ZSL) tackles the novel class recognition problem by transferring semantic knowledge from seen classes to unseen ones. Semantic knowledge is typically represented by attribute descriptions shared between different classes, which act as strong priors for localizing object attributes that represent discriminative region features, enabling significant and sufficient visual-semantic interaction for advancing ZSL. Existing attention-based models have struggled to learn inferior region features in a single image by solely using unidirectional attention, which ignore the transferable and discriminative attribute localization of visual features for representing the key semantic knowledge for effective knowledge transfer in ZSL. In this paper, we propose a cross attribute-guided Transformer network, termed TransZero++, to refine visual features and learn accurate attribute localization for key semantic knowledge representations in ZSL. Specifically, TransZero++ employs an attribute $\rightarrow$ visual Transformer sub-net (AVT) and a visual $\rightarrow$ attribute Transformer sub-net (VAT) to learn attribute-based visual features and visual-based attribute features, respectively. By further introducing feature-level and prediction-level semantical collaborative losses, the two attribute-guided transformers teach each other to learn semantic-augmented visual embeddings for key semantic knowledge representations via semantical collaborative learning. Finally, the semantic-augmented visual embeddings learned by AVT and VAT are fused to conduct desirable visual-semantic interaction cooperated with class semantic vectors for ZSL classification. Extensive experiments show that TransZero++ achieves the new state-of-the-art results on three golden ZSL benchmarks and on the large-scale ImageNet dataset. The project website is available at: https://shiming-chen.github.io/TransZero-pp/TransZero-pp.html .