TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint Prediction in Aerial Scenes
- 8 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62 (01962892) , 1-16
Abstract
As drone technology advances, using unmanned aerial vehicles for aerial surveys has become the dominant trend in modern low-altitude remote sensing. The surge in aerial video data necessitates accurate prediction for future scenarios and motion states of the interested target, particularly in applications like traffic management and disaster response. Existing video prediction methods focus solely on predicting future scenes (video frames), suffering from the neglect of explicitly modeling target’s motion states, which is crucial for aerial video interpretation. To address this issue, we introduce a novel task called Target-Aware Aerial Video Prediction, aiming to simultaneously predict future scenes and motion states of the target. Further, we design a model specifically for this task, named TAFormer, which provides a unified modeling approach for both video and target motion states. Specifically, we introduce Spatiotemporal Attention (STA), which decouples the learning of video dynamics into spatial static attention and temporal dynamic attention, effectively modeling the scene appearance and motion. Additionally, we design an Information Sharing Mechanism (ISM), which elegantly unifies the modeling of video and target motion by facilitating information interaction through two sets of messenger tokens. Moreover, to alleviate the difficulty of distinguishing targets in blurry predictions, we introduce Target-Sensitive Gaussian Loss (TSGL), enhancing the model’s sensitivity to both target’s position and content. Extensive experiments on UAV123VP and VisDroneVP (derived from single-object tracking datasets) demonstrate the exceptional performance of TAFormer in target-aware video prediction, showcasing its adaptability to the additional requirements of aerial video interpretation for target awareness.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62171436, 62331027)
- Key Deployment Program of the Chinese Academy of Sciences (KGFZD-145-23-18)
This publication has 50 references indexed in Scilit:
- SimVP: Simpler yet Better Video Prediction2022
- Behavior Prediction of Traffic Actors for Intelligent Vehicle Using Artificial Intelligence Techniques: A ReviewIEEE Access, 2021
- TPCN: Temporal Point Cloud Networks for Motion Forecasting2021
- Future Video Synthesis With Object Motion Prediction2020
- Memory in Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity From Spatiotemporal Dynamics2019
- Rules of the Road: Predicting Driving Behavior With a Convolutional Model of Semantic Interactions2019
- Convolutional Neural Network for Trajectory Prediction2019
- Folded Recurrent Neural Networks for Future Video Prediction2018
- Squeeze-and-Excitation Networks2018
- Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net2018