Flexible Multitask Learning With Factorized Diffusion Policy
Open Access
- 13 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (4) , 4697-4704
Abstract
Multitask learning poses significant challenges due to the highly multimodal and diverse nature of robot action distributions. However, effectively fitting policies to these complex task distributions is often difficult, and existing monolithic models often underfit the action distribution and lack the flexibility required for efficient adaptation. We introduce a novel modular diffusion policy framework that factorizes complex action distributions into a composition of specialized diffusion models, each capturing a distinct sub-mode of the behavior space for a more effective overall policy. In addition, this modular structure enables flexible policy adaptation to new tasks by adding or fine-tuning components, which inherently mitigates catastrophic forgetting. Empirically, across both simulation and real-world robotic manipulation settings, we illustrate how our method consistently outperforms strong modular and monolithic baselines.Keywords
All Related Versions
Funding Information
- Kempner Institute at Harvard University
This publication has 15 references indexed in Scilit:
- Hybrid Diffusion for Simultaneous Symbolic and Continuous Planning_supp1-3664616.mp42026
- Learning Coordinated Bimanual Manipulation Policies Using State Diffusion and Inverse Dynamics Models2025
- Diffusion Soup: Model Merging for Text-to-Image Diffusion Models2024
- Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models2023
- Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models2023
- SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion2023
- Compositional Visual Generation with Composable Diffusion Models2022
- Robot Sound Interpretation: Combining Sight and Sound in Learning-Based Control2020
- RLBench: The Robot Learning Benchmark & Learning EnvironmentIEEE Robotics and Automation Letters, 2020
- Learning modular neural network policies for multi-task and multi-robot transfer2017