DNAct: Diffusion Guided Multi-Task 3D Policy Learning
Authors:
Top Cited Papers
- 19 October 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9464-9471
Abstract
This paper presents DNAct, a language-conditioned multi-task policy framework that integrates neural rendering pre-training and diffusion training to enforce multi-modality learning in action sequence spaces. To learn a generalizable multi-task policy with few demonstrations, the pre-training phase of DNAct leverages neural rendering to distill 2D semantic features from foundation models such as Stable Diffusion to a 3D space, which provides a comprehensive semantic understanding regarding the scene. Consequently, it allows various applications for challenging robotic tasks requiring rich 3D semantics and accurate geometry. Furthermore, we introduce a novel approach utilizing diffusion training to learn a vision and language feature that encapsulates the inherent multi-modality in the multi-task demonstrations. By reconstructing the action sequences from different tasks via the diffusion process, the model is capable of distinguishing different modalities and thus improving the robustness and the generalizability of the learned representation. DNAct significantly surpasses SOTA NeRF-based multi-task manipulation approaches with over 30% improvement in success rate. Videos are available on dnact.github.ioKeywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Diffusion-based Generation, Optimization, and Planning in 3D Scenes2023
- Transformers as Meta-learners for Implicit Neural Representations2022
- Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- IBRNet: Learning Multi-View Image-Based Rendering2021
- Learning Continuous Image Representation with Local Implicit Image Function2021
- 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation2019
- DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation2019
- Occupancy Networks: Learning 3D Reconstruction in Function Space2019
- SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels2013