Jiahua Dong
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Jiahua Dong has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
Yu-Xiong Wang
University of Illinois Urbana-Champaign
2 shared publications- Yu-Xiong Wang
University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign
2 shared publications - Aaron Saxton
National Center for Supercomputing Applications, University of Illinois Urbana-Champaign, Champaign, IL 61801, USA
1 shared publication - Albert Bode
National Center for Supercomputing Applications, University of Illinois Urbana-Champaign, Champaign, IL 61801, USA
1 shared publication - Dou Hoon Kwark
University of Illinois Urbana-Champaign
1 shared publication - Jipeng Lyu
University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign
1 shared publication
Latest Publications
- Open Access
Capturing Visual Environment Structure Correlates with Control Performance
The choice of visual representation is key to scaling generalist robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state -- including geometry, object structure, and physical attributes -- from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode the latent physical state of the environment is a promising objective for control.
Open AccessCapturing Visual Environment Structure Correlates with Control Performance
The choice of visual representation is key to scaling generalist robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state -- including geometry, object structure, and physical attributes -- from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode the latent physical state of the environment is a promising objective for control.
- Open Access
3DGS-Drag: Dragging Gaussians for Intuitive Point-Based 3D Editing
The transformative potential of 3D content creation has been progressively unlocked through advancements in generative models. Recently, intuitive drag editing with geometric changes has attracted significant attention in 2D editing yet remains challenging for 3D scenes. In this paper, we introduce 3DGS-Drag -- a point-based 3D editing framework that provides efficient, intuitive drag manipulation of real 3D scenes. Our approach bridges the gap between deformation-based and 2D-editing-based 3D editing methods, addressing their limitations to geometry-related content editing. We leverage two key innovations: deformation guidance utilizing 3D Gaussian Splatting for consistent geometric modifications and diffusion guidance for content correction and visual quality enhancement. A progressive editing strategy further supports aggressive 3D drag edits. Our method enables a wide range of edits, including motion change, shape adjustment, inpainting, and content extension. Experimental results demonstrate the effectiveness of 3DGS-Drag in various scenes, achieving state-of-the-art performance in geometry-related 3D content editing. Notably, the editing is efficient, taking 10 to 20 minutes on a single RTX 4090 GPU.
Open Access3DGS-Drag: Dragging Gaussians for Intuitive Point-Based 3D Editing
The transformative potential of 3D content creation has been progressively unlocked through advancements in generative models. Recently, intuitive drag editing with geometric changes has attracted significant attention in 2D editing yet remains challenging for 3D scenes. In this paper, we introduce 3DGS-Drag -- a point-based 3D editing framework that provides efficient, intuitive drag manipulation of real 3D scenes. Our approach bridges the gap between deformation-based and 2D-editing-based 3D editing methods, addressing their limitations to geometry-related content editing. We leverage two key innovations: deformation guidance utilizing 3D Gaussian Splatting for consistent geometric modifications and diffusion guidance for content correction and visual quality enhancement. A progressive editing strategy further supports aggressive 3D drag edits. Our method enables a wide range of edits, including motion change, shape adjustment, inpainting, and content extension. Experimental results demonstrate the effectiveness of 3DGS-Drag in various scenes, achieving state-of-the-art performance in geometry-related 3D content editing. Notably, the editing is efficient, taking 10 to 20 minutes on a single RTX 4090 GPU.
- Open Access
SCas4D: Structural Cascaded Optimization for Boosting Persistent 4D Novel View Synthesis
Persistent dynamic scene modeling for tracking and novel-view synthesis remains challenging due to the difficulty of capturing accurate deformations while maintaining computational efficiency. We propose SCas4D, a cascaded optimization framework that leverages structural patterns in 3D Gaussian Splatting for dynamic scenes. The key idea is that real-world deformations often exhibit hierarchical patterns, where groups of Gaussians share similar transformations. By progressively refining deformations from coarse part-level to fine point-level, SCas4D achieves convergence within 100 iterations per time frame and produces results comparable to existing methods with only one-twentieth of the training iterations. The approach also demonstrates effectiveness in self-supervised articulated object segmentation, novel view synthesis, and dense point tracking tasks.
Open AccessSCas4D: Structural Cascaded Optimization for Boosting Persistent 4D Novel View Synthesis
Persistent dynamic scene modeling for tracking and novel-view synthesis remains challenging due to the difficulty of capturing accurate deformations while maintaining computational efficiency. We propose SCas4D, a cascaded optimization framework that leverages structural patterns in 3D Gaussian Splatting for dynamic scenes. The key idea is that real-world deformations often exhibit hierarchical patterns, where groups of Gaussians share similar transformations. By progressively refining deformations from coarse part-level to fine point-level, SCas4D achieves convergence within 100 iterations per time frame and produces results comparable to existing methods with only one-twentieth of the training iterations. The approach also demonstrates effectiveness in self-supervised articulated object segmentation, novel view synthesis, and dense point tracking tasks.
- Open Access
- 13 November 2024
- Published by MDPI AG in Geosciences
- Vol. 14 (11) , 305
Accurate Feature Extraction from Historical Geologic Maps Using Open-Set Segmentation and Detection
This study presents a novel AI method for extracting polygon and point features from historical geologic maps, representing a pivotal step for assessing the mineral resources needed for energy transition. Our innovative method involves using map units in the legends as prompts for one-shot segmentation and detection in geological feature extraction. The model, integrated with a human-in-the-loop system, enables geologists to refine results efficiently, combining the power of AI with expert oversight. Tested on geologic maps annotated by USGS and DARPA for the AI4CMA DARPA Challenge, our approach achieved a median F1 score of 0.91 for polygon feature segmentation and 0.73 for point feature detection when such features had abundant annotated data, outperforming current benchmarks. By efficiently and accurately digitizing historical geologic map, our method promises to provide crucial insights for responsible policymaking and effective resource management in the global energy transition.
- 13 November 2024
- Published by MDPI AG in Geosciences
- Vol. 14 (11) , 305
Open AccessAccurate Feature Extraction from Historical Geologic Maps Using Open-Set Segmentation and Detection
This study presents a novel AI method for extracting polygon and point features from historical geologic maps, representing a pivotal step for assessing the mineral resources needed for energy transition. Our innovative method involves using map units in the legends as prompts for one-shot segmentation and detection in geological feature extraction. The model, integrated with a human-in-the-loop system, enables geologists to refine results efficiently, combining the power of AI with expert oversight. Tested on geologic maps annotated by USGS and DARPA for the AI4CMA DARPA Challenge, our approach achieved a median F1 score of 0.91 for polygon feature segmentation and 0.73 for point feature detection when such features had abundant annotated data, outperforming current benchmarks. By efficiently and accurately digitizing historical geologic map, our method promises to provide crucial insights for responsible policymaking and effective resource management in the global energy transition.
- Open Access
ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is publicly available.
Open AccessViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is publicly available.