Controllable Weather Synthesis and Removal with Video Diffusion Models

Authors:
  • CLChih-Hao Lin
    Chih-Hao Lin
  • ZWZian Wang
    Zian Wang
  • RLRuofan Liang
    Ruofan Liang
  • YZYuxuan Zhang
    Yuxuan Zhang
  • SFSanja Fidler
    Sanja Fidler
  • SWShenlong Wang
    Shenlong Wang
  • ZGZan Gojcic
    Zan Gojcic

Abstract
Generating realistic and controllable weather effects in videos is valuable for many applications. Physics-based weather simulation requires precise reconstructions that are hard to scale to in-the-wild videos, while current video editing often lacks realism and control. In this work, we introduce WeatherWeaver, a video diffusion model that synthesizes diverse weather effects -- including rain, snow, fog, and clouds -- directly into any input video without the need for 3D modeling. Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability. To overcome the scarcity of paired training data, we propose a novel data strategy combining synthetic videos, generative image editing, and auto-labeled real-world videos. Extensive evaluations show that our method outperforms state-of-the-art methods in weather simulation and removal, providing high-quality, physically plausible, and scene-identity-preserving results over various real-world videos.