LongShortNet: Exploring Temporal and Semantic Features Fusion in Streaming Perception

Authors:
  • CLChenyang Li
    Chenyang Li
  • ZCZhi-Qi Cheng
    Zhi-Qi Cheng
  • JHJun-Yan He
    Jun-Yan He
  • PLPengyu Li
    Pengyu Li
  • BLBin Luo
    Bin Luo
  • HCHan-Yuan Chen
    Han-Yuan Chen
  • YGYifeng Geng
    Yifeng Geng
  • JLJin-Peng Lan
    Jin-Peng Lan
  • XXXuansong Xie
    Xuansong Xie

Abstract
Streaming perception is a task of reporting the current state of autonomous driving, which coherently considers the latency and accuracy of autopilot systems. However, the existing streaming perception only uses the current and adjacent two frames as input for learning the movement patterns, which cannot model actual complex scenes, resulting in failed detection results. To solve this problem, we propose an end-to-end dual-path network dubbed LongShortNet, which captures long-term temporal motion and calibrates it with short-term spatial semantics for real-time perception. Moreover, we investigate a Long-Short Fusion Module (LSFM) to explore spatiotemporal feature fusion, which is the first work to extend long-term temporal in streaming perception. We evaluate the proposed LongShortNet and compare it with existing methods on the benchmark dataset Argoverse-HD. The results demonstrate that the proposed LongShortNet outperforms the other state-of-the-art methods with almost no extra computational cost.