Sophia-in-Audition: Virtual Production with a Robot Performer
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 11184-11193
Abstract
We present Sophia-in-Audition (SiA), a new frontier in virtual production, by employing the humanoid robot Sophia within an UltraStage environment composed of a controllable lighting dome coupled with multiple cameras. We demonstrate Sophia's capability to replicate iconic film segments, follow real performers, and perform a variety of motions and expressions, showcasing her versatility as a virtual actor. Key to this process is the integration of facial motion transfer algorithms and the UltraStage's controllable lighting and multi-camera setup, enabling dynamic performances that align with the director's vision. Our comprehensive user studies indicate positive audience reception towards Sophia's performances, highlighting her potential to reduce the uncanny valley effect in virtual acting. Additionally, the immersive lighting in dynamic clips was highly rated for its naturalness and its ability to mirror professional film standards. The paper presents a first-of-its-kind multi-view robot performance video dataset with dynamic lighting, offering valuable insights for future enhancements in humanoid robotic performers and virtual production techniques. This research contributes significantly to the field by presenting a unique virtual production setup, developing tools for sophisticated performance control, and providing a comprehensive dataset and user study analysis for diverse applications.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- DiffusionLight: Light Probes for Free by Painting a Chrome Ball2024
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- ABO: Dataset and Benchmarks for Real-World 3D Object Understanding2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Plenoxels: Radiance Fields without Neural Networks2022
- PlenOctrees for Real-time Rendering of Neural Radiance Fields2021
- pixelNeRF: Neural Radiance Fields from One or Few Images2021
- StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images2020
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization2019
- Mirror MoCap: Automatic and efficient capture of dense 3D facial motion parameters from videoThe Visual Computer, 2005