Cheng Sun
Research Subjects
Cheng Sun has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Chu-Song Chen
Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan
1 shared publication Jaesung Choe
NVIDIA
1 shared publicationSheng-Yu Huang
NVIDIA
1 shared publication- Wei-Hua Li
National Taiwan University
1 shared publication - Yi-Chia Chen
National Taiwan University
1 shared publication Yu-Chiang Frank Wang
NVIDIA
1 shared publication
Latest Publications
- Open Access
OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding
We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.
Open AccessOpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding
We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.
- 1 November 2024
- Published by Springer Nature
- p. 323-340
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation
- 1 November 2024
- Published by Springer Nature
- p. 323-340
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation