ITEM: Immersive Telepresence for Entertainment and Meetings—A Practical Approach
- 2 December 2014
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 9 (3) , 546-561
Abstract
This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive (TI) application scenarios. In addition, we also present a few key technologies, i.e., fast object-based video coding for real world data and spatialized audio capture and 3-D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g., a webcam or a color plus depth camera, an optional microphone array, a laptop/desktop connected to the Internet) for TI communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.Keywords
All Related Versions
Funding Information
- research grant for the Human Sixth Sense Programme at the Advanced Digital Sciences Center from Singapores Agency for Science Technology and Research ASTAR
This publication has 28 references indexed in Scilit:
- Virtual Mirror Rendering With Stationary RGB-D Cameras and Stored 3-D BackgroundIEEE Transactions on Image Processing, 2013
- Towards next generation 3D teleconferencing systems2012
- Hello, is grandma there? let's read! StoryVisit2011
- Exploring spatialized audio & video for distributed conversations2010
- Piecewise-consistent color mappings of images acquired under various conditions2009
- Efficient Video Transcoding from H.263 to H.264/AVC Standard with Enhanced Rate ControlEURASIP Journal on Advances in Signal Processing, 2006
- Beamformer performance with acoustic vector sensors in airThe Journal of the Acoustical Society of America, 2006
- Understanding performance in coliseum, an immersive videoconferencing systemACM Transactions on Multimedia Computing, Communications, and Applications, 2005
- Distributed meetings2002
- A Standard-Compliant Virtual Meeting System with Active Video Object TrackingEURASIP Journal on Advances in Signal Processing, 2002