DiffusionCinema: Text-to-Aerial Cinematography
Preprint
Abstract
We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories for cinematic video recording. Instead of manually piloting the drone, the user simply describes the desired shot (e.g., "orbit around me slowly from the right and reveal the background waterfall"). Our system encodes the prompt along with an initial visual snapshot from the onboard camera, and a diffusion model samples plausible spatio-temporal motion plans that satisfy both the scene geometry and shot semantics. The generated flight trajectory is then executed autonomously by the UAV to record smooth, repeatable video clips that match the prompt. User evaluation using NASA-TLX showed a significantly lower overall workload with our interface (M = 21.6) compared to a traditional remote controller (M = 58.1), demonstrating a substantial reduction in perceived effort. Mental demand (M = 11.5 vs. 60.5) and frustration (M = 14.0 vs. 54.5) were also markedly lower for our system, confirming clear usability advantages in autonomous text-driven flight control. This project demonstrates a new interaction paradigm: text-to-cinema flight, where diffusion models act as the "creative operator" converting story intentions directly into aerial motion.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- LLM-based UAV Path Planning for Autonomous and Adaptive Industry SystemsACM Transactions on Autonomous and Adaptive Systems, 2025
- AgilePilot: DRL-Based Drone Agent for Real-Time Motion Planning in Dynamic Environments by Leveraging Object Detection2025
- Autonomous Drone Racing: A SurveyIEEE Transactions on Robotics, 2024
- Artificial Intelligence Approaches for UAV Navigation: Recent Advances and Future ChallengesIEEE Access, 2022
- ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAMIEEE Transactions on Robotics, 2021
- Human-Piloted Drone Racing: Visual Processing and ControlIEEE Robotics and Automation Letters, 2021
- OpenVINS: A Research Platform for Visual-Inertial Estimation2020
- Autonomous UAV CinematographyACM Computing Surveys, 2019
- Challenges in Autonomous UAV Cinematography: An Overview2018
- Nasa-Task Load Index (NASA-TLX); 20 Years LaterProceedings of the Human Factors and Ergonomics Society Annual Meeting, 2006