Viewport prediction with cross modal multiscale transformer for 360° video streaming
Open Access
- 19 August 2025
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 15 (1) , 1-12
Abstract
In the realm of immersive video technologies, efficient 360° video streaming remains a challenge due to the high bandwidth requirements and the dynamic nature of user viewports. Most existing approaches neglect the dependencies between different modalities, and personal preferences are rarely considered. These limitations lead to inconsistent prediction performance. Here, we present a novel viewport prediction model leveraging a Cross Modal Multiscale Transformer (CMMST) that integrates user trajectory and video saliency features across different scales. Our approach outperforms baseline methods, maintaining high precision even with extended prediction intervals. By harnessing the Cross Modal attention mechanisms, CMMST captures intricate user preferences and viewing patterns, offering a promising solution for adaptive streaming in virtual reality and other immersive platforms. The code of this work is available at https://github.com/bbgua85776540/CMMST.Keywords
This publication has 19 references indexed in Scilit:
- A self-attention model for viewport prediction based on distance constraintThe Visual Computer, 2023
- LSTM-based Viewport Prediction for Immersive Video Systems2023
- A Survey on the Metaverse: The State-of-the-Art, Technologies, Applications, and ChallengesIEEE Internet of Things Journal, 2023
- Transformer-based Long-Term Viewport Prediction in 360° Video: Scanpath is All You Need2021
- Multiscale Vision Transformers2021
- ATSal: An Attention Based Architecture for Saliency Prediction in 360$^\circ $ Videos2021
- A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and OpportunitiesIEEE Communications Surveys & Tutorials, 2020
- Viewport-Dependent Saliency Prediction in 360° VideoIEEE Transactions on Multimedia, 2020
- Compressed domain video processing for tile based panoramic streaming using HEVC2015
- The MPEG-DASH Standard for Multimedia Streaming Over the InternetIEEE MultiMedia, 2011