OctFormer: Octree-based Transformers for 3D Point Clouds
Top Cited Papers
Open Access
- 26 July 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 42 (4) , 1-11
Abstract
We propose octree-based transformers, named OctFormer, for 3D point cloud learning. OctFormer can not only serve as a general and effective backbone for 3D point cloud segmentation and object detection but also have linear complexity and is scalable for large-scale point clouds. The key challenge in applying transformers to point clouds is reducing the quadratic, thus overwhelming, computation complexity of attentions. To combat this issue, several works divide point clouds into non-overlapping windows and constrain attentions in each local window. However, the point number in each window varies greatly, impeding the efficient execution on GPU. Observing that attentions are robust to the shapes of local windows, we propose a novel octree attention, which leverages sorted shuffled keys of octrees to partition point clouds into local windows containing a fixed number of points while permitting shapes of windows to change freely. And we also introduce dilated octree attention to expand the receptive field further. Our octree attention can be implemented in 10 lines of code with open-sourced libraries and runs 17 times faster than other point cloud attentions when the point number exceeds 200 k. Built upon the octree attention, OctFormer can be easily scaled up and achieves state-of-the-art performances on a series of 3D semantic segmentation and 3D object detection benchmarks, surpassing previous sparse-voxel-based CNNs and point cloud transformers in terms of both efficiency and effectiveness. Notably, on the challenging ScanNet200 dataset, OctFormer outperforms sparse-voxel-based CNNs by 7.3 in mIoU. Our code and trained models are available at https://wang-ps.github.io/octformer.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- A Unified Point-Based Framework for 3D Segmentation2019
- Octree Guided CNN With Spherical Kernels for 3D Point Clouds2019
- Point convolutional neural networks by extension operatorsACM Transactions on Graphics, 2018
- 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks2018
- O-CNNACM Transactions on Graphics, 2017
- Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs2017
- VoxNet: A 3D Convolutional Neural Network for real-time object recognition2015
- 3D ShapeNets: A deep representation for volumetric shapes2015
- SUN RGB-D: A RGB-D scene understanding benchmark suite2015
- Octrees for faster isosurface generationACM Transactions on Graphics, 1992