Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos
Authors:
- Hou-Ning HuHHHou-Ning Hu
- Yen-Chen LinYLYen-Chen Lin
- Ming-Yu LiuMLMing-Yu Liu
- Hsien-Tzu ChengHCHsien-Tzu Cheng
- Yung-Ju ChangYCYung-Ju Chang
- Min SunMSMin Sun
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 1396-1405
Abstract
Watching a 360° sports video requires a viewer to continuously select a viewing angle, either through a sequence of mouse clicks or head movements. To relieve the viewer from this “360 piloting” task, we propose “deep 360 pilot” - a deep learning-based agent for piloting through 360° sports videos automatically. At each frame, the agent observes a panoramic image and has the knowledge of previously selected viewing angles. The task of the agent is to shift the current viewing angle (i.e. action) to the next preferred one (i.e., goal). We propose to directly learn an online policy of the agent from data. Specifically, we leverage a state-of-the-art object detector to propose a few candidate objects of interest (yellow boxes in Fig. 1). Then, a recurrent neural network is used to select the main object (green dash boxes in Fig. 1). Given the main object and previously selected viewing angles, our method regresses a shift in viewing angle to move to the next one. We use the policy gradient technique to jointly train our pipeline, by minimizing: (1) a regression loss measuring the distance between the selected and ground truth viewing angles, (2) a smoothness loss encouraging smooth transition in viewing angle, and (3) maximizing an expected reward offocusing on a foreground object. To evaluate our method, we built a new 360-Sports video dataset consisting offive sports domains. We trained domain-specific agents and achieved the best performance on viewing angle selection accuracy and users' preference compared to [53] and other baselines.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- Exploiting Surroundedness for Saliency Detection: A Boolean Map ApproachIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- First-person hyper-lapse videosACM Transactions on Graphics, 2014
- Cliplets2012
- Learning to Recognize Daily Actions Using Gaze2012
- Clustering of Gaze During Dynamic Scene Viewing is Predicted by MotionCognitive Computation, 2010
- Static and space-time visual saliency detection by self-resemblanceJournal of Vision, 2009
- Schematic storyboarding for video visualization and editing2006
- The virtual cinematographer1996
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992