Matthew Tancik
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Matthew Tancik has a total of 25 co-authors. Here are the top co-authors by number of shared publications:
Ramesh Raskar
Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA
5 shared publications- Angjoo Kanazawa
UC Berkeley
3 shared publications - Guy Satat
MIT Media Lab 75 Amherst st Cambridge MA USA
3 shared publications - Barmak Heshmat
Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA
2 shared publications - Ethan Weber
University of California Berkeley, United States of America
2 shared publications Guy Satat
MIT Media Lab
2 shared publications
Latest Publications
- Top Cited Papers
- 16 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 35, 21530-21539
GARField: Group Anything with Radiance Fields
Grouping is inherently ambiguous due to the multiple levels of granularity in which one can decompose a sceneshould the wheels of an excavator be considered separate or part of the whole? We propose Group Anything with Radiance Fields (GARField), an approach for decomposing 3D scenes into a hierarchy of semantically meaningful groups from posed image inputs. To do this we embrace group ambiguity through physical scale: by optimizing a scaleconditioned 3D affinity feature field, a point in the world can belong to different groups of different sizes. We optimize this field from a set of 2D masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine hierarchy, using scale to consistently fuse conflicting masks from different viewpoints. From this field we can derive a hierarchy of possible groupings via automatic tree construction or user interaction. We evaluate GARField on a variety of in-the-wild scenes and find it effectively extracts groups at many levels: clusters of objects, objects, and various subparts. GARField inherently represents multi-view consistent groupings and produces higher fidelity groups than the input SAM masks. GARField's hierarchical grouping could have exciting downstream applications such as 3D asset extraction or dynamic scene understanding. Project site: https://www.garfield.studio/
- 16 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 35, 21530-21539
Top Cited PapersGARField: Group Anything with Radiance Fields
Grouping is inherently ambiguous due to the multiple levels of granularity in which one can decompose a sceneshould the wheels of an excavator be considered separate or part of the whole? We propose Group Anything with Radiance Fields (GARField), an approach for decomposing 3D scenes into a hierarchy of semantically meaningful groups from posed image inputs. To do this we embrace group ambiguity through physical scale: by optimizing a scaleconditioned 3D affinity feature field, a point in the world can belong to different groups of different sizes. We optimize this field from a set of 2D masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine hierarchy, using scale to consistently fuse conflicting masks from different viewpoints. From this field we can derive a hierarchy of possible groupings via automatic tree construction or user interaction. We evaluate GARField on a variety of in-the-wild scenes and find it effectively extracts groups at many levels: clusters of objects, objects, and various subparts. GARField inherently represents multi-view consistent groupings and produces higher fidelity groups than the input SAM masks. GARField's hierarchical grouping could have exciting downstream applications such as 3D asset extraction or dynamic scene understanding. Project site: https://www.garfield.studio/
- Top Cited PapersOpen Access
- 23 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Nerfstudio: A Modular Framework for Neural Radiance Field Development
- 23 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Top Cited PapersOpen AccessNerfstudio: A Modular Framework for Neural Radiance Field Development
- Open Access
The One Where They Reconstructed 3D Humans and Environments in TV Shows
TV shows depict a wide variety of human behaviors and have been studied extensively for their potential to be a rich source of data for many applications. However, the majority of the existing work focuses on 2D recognition tasks. In this paper, we make the observation that there is a certain persistence in TV shows, i.e., repetition of the environments and the humans, which makes possible the 3D reconstruction of this content. Building on this insight, we propose an automatic approach that operates on an entire season of a TV show and aggregates information in 3D; we build a 3D model of the environment, compute camera information, static 3D scene structure and body scale information. Then, we demonstrate how this information acts as rich 3D context that can guide and improve the recovery of 3D human pose and position in these environments. Moreover, we show that reasoning about humans and their environment in 3D enables a broad range of downstream applications: re-identification, gaze estimation, cinematography and image editing. We apply our approach on environments from seven iconic TV shows and perform an extensive evaluation of the proposed system.
Open AccessThe One Where They Reconstructed 3D Humans and Environments in TV Shows
TV shows depict a wide variety of human behaviors and have been studied extensively for their potential to be a rich source of data for many applications. However, the majority of the existing work focuses on 2D recognition tasks. In this paper, we make the observation that there is a certain persistence in TV shows, i.e., repetition of the environments and the humans, which makes possible the 3D reconstruction of this content. Building on this insight, we propose an automatic approach that operates on an entire season of a TV show and aggregates information in 3D; we build a 3D model of the environment, compute camera information, static 3D scene structure and body scale information. Then, we demonstrate how this information acts as rich 3D context that can guide and improve the recovery of 3D human pose and position in these environments. Moreover, we show that reasoning about humans and their environment in 3D enables a broad range of downstream applications: re-identification, gaze estimation, cinematography and image editing. We apply our approach on environments from seven iconic TV shows and perform an extensive evaluation of the proposed system.
- 13 August 2018
- Published by Springer Nature in Nature Photonics
- Vol. 12 (9) , 560-566
Photography optics in the time dimension
- 13 August 2018
- Published by Springer Nature in Nature Photonics
- Vol. 12 (9) , 560-566
Photography optics in the time dimension
- Top Cited Papers
- 1 May 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Towards photography through realistic fog
Imaging through fog has important applications in industries such as self-driving cars, augmented driving, airplanes, helicopters, drones and trains. Here we show that time profiles of light reflected from fog have a distribution (Gamma) that is different from light reflected from objects occluded by fog (Gaussian). This helps to distinguish between background photons reflected from the fog and signal photons reflected from the occluded object. Based on this observation, we recover reflectance and depth of a scene obstructed by dense, dynamic, and heterogeneous fog. For practical use cases, the imaging system is designed in optical reflection mode with minimal footprint and is based on LIDAR hardware. Specifically, we use a single photon avalanche diode (SPAD) camera that time-tags individual detected photons. A probabilistic computational framework is developed to estimate the fog properties from the measurement itself, without prior knowledge. Other solutions are based on radar that suffers from poor resolution (due to the long wavelength), or on time gating that suffers from low signal-to-noise ratio. The suggested technique is experimentally evaluated in a wide range of fog densities created in a fog chamber It demonstrates recovering objects 57cm away from the camera when the visibility is 37cm. In that case it recovers depth with a resolution of 5cm and scene reflectance with an improvement of 4dB in PSNR and 3.4× reconstruction quality in SSIM over time gating techniques.
- 1 May 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Top Cited PapersTowards photography through realistic fog
Imaging through fog has important applications in industries such as self-driving cars, augmented driving, airplanes, helicopters, drones and trains. Here we show that time profiles of light reflected from fog have a distribution (Gamma) that is different from light reflected from objects occluded by fog (Gaussian). This helps to distinguish between background photons reflected from the fog and signal photons reflected from the occluded object. Based on this observation, we recover reflectance and depth of a scene obstructed by dense, dynamic, and heterogeneous fog. For practical use cases, the imaging system is designed in optical reflection mode with minimal footprint and is based on LIDAR hardware. Specifically, we use a single photon avalanche diode (SPAD) camera that time-tags individual detected photons. A probabilistic computational framework is developed to estimate the fog properties from the measurement itself, without prior knowledge. Other solutions are based on radar that suffers from poor resolution (due to the long wavelength), or on time gating that suffers from low signal-to-noise ratio. The suggested technique is experimentally evaluated in a wide range of fog densities created in a fog chamber It demonstrates recovering objects 57cm away from the camera when the visibility is 37cm. In that case it recovers depth with a resolution of 5cm and scene reflectance with an improvement of 4dB in PSNR and 3.4× reconstruction quality in SSIM over time gating techniques.
- 1 January 2018
- Published by Optica Publishing Group
- p. IW2B.6
Data-Driven Non-Line-of-Sight Imaging With A Traditional Camera
Non-line-of-right (NLOS) imaging has recently been demonstrated using traditional cameras. Here we present a data-driven technique for NLOS imaging. We experimentally analyze the role of scene geometry and clutter on reconstruction quality.
- 1 January 2018
- Published by Optica Publishing Group
- p. IW2B.6
Data-Driven Non-Line-of-Sight Imaging With A Traditional Camera
Non-line-of-right (NLOS) imaging has recently been demonstrated using traditional cameras. Here we present a data-driven technique for NLOS imaging. We experimentally analyze the role of scene geometry and clutter on reconstruction quality.
- 1 January 2018
- Published by Optica Publishing Group
- p. AF3Q.4
Introduction to Time-folded Optics; Spatial Compression of Paraxial Optics Using Time
We exploit the high time resolution of ultrafast imaging sensors to fold and modify the imaging optics of the camera. We demonstrate an order of magnitude compression in lens–sensor distance using a time-folded design.
- 1 January 2018
- Published by Optica Publishing Group
- p. AF3Q.4
Introduction to Time-folded Optics; Spatial Compression of Paraxial Optics Using Time
We exploit the high time resolution of ultrafast imaging sensors to fold and modify the imaging optics of the camera. We demonstrate an order of magnitude compression in lens–sensor distance using a time-folded design.
- Top Cited PapersOpen Access
- 12 July 2017
- Published by Optica Publishing Group in Optics Express
- Vol. 25 (15) , 17466-17479
Object classification through scattering media with deep learning on time resolved measurement
We demonstrate an imaging technique that allows identification and classification of objects hidden behind scattering media and is invariant to changes in calibration parameters within a training range. Traditional techniques to image through scattering solve an inverse problem and are limited by the need to tune a forward model with multiple calibration parameters (like camera field of view, illumination position etc.). Instead of tuning a forward model and directly inverting the optical scattering, we use a data driven approach and leverage convolutional neural networks (CNN) to learn a model that is invariant to calibration parameters variations within the training range and nearly invariant beyond that. This effectively allows robust imaging through scattering conditions that is not sensitive to calibration. The CNN is trained with a large synthetic dataset generated with a Monte Carlo (MC) model that contains random realizations of major calibration parameters. The method is evaluated with a time-resolved camera and multiple experimental results are provided including pose estimation of a mannequin hidden behind a paper sheet with 23 correct classifications out of 30 tests in three poses (76.6% accuracy on real-world measurements). This approach paves the way towards real-time practical non line of sight (NLOS) imaging applications.
- 12 July 2017
- Published by Optica Publishing Group in Optics Express
- Vol. 25 (15) , 17466-17479
Top Cited PapersOpen AccessObject classification through scattering media with deep learning on time resolved measurement
We demonstrate an imaging technique that allows identification and classification of objects hidden behind scattering media and is invariant to changes in calibration parameters within a training range. Traditional techniques to image through scattering solve an inverse problem and are limited by the need to tune a forward model with multiple calibration parameters (like camera field of view, illumination position etc.). Instead of tuning a forward model and directly inverting the optical scattering, we use a data driven approach and leverage convolutional neural networks (CNN) to learn a model that is invariant to calibration parameters variations within the training range and nearly invariant beyond that. This effectively allows robust imaging through scattering conditions that is not sensitive to calibration. The CNN is trained with a large synthetic dataset generated with a Monte Carlo (MC) model that contains random realizations of major calibration parameters. The method is evaluated with a time-resolved camera and multiple experimental results are provided including pose estimation of a mannequin hidden behind a paper sheet with 23 correct classifications out of 30 tests in three poses (76.6% accuracy on real-world measurements). This approach paves the way towards real-time practical non line of sight (NLOS) imaging applications.
- 20 March 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 3 (3) , 398-407
Lensless Imaging With Compressive Ultrafast Sensing
Lensless imaging is an important and challenging problem. One notable solution to lensless imaging is a single-pixel camera that benefits from ideas central to compressive sampling. However, traditional single-pixel cameras require many illumination patterns that result in a long acquisition process. Here, we present a method for lensless imaging based on compressive ultrafast sensing. Each sensor acquisition is encoded with a different illumination pattern and produces a time series where time is a function of the photon's origin in the scene. Currently available hardware with picosecond time resolution enables time tagging photons as they arrive to an omnidirectional sensor. This allows lensless imaging with significantly fewer patterns compared to regular single-pixel imaging. To that end, we develop a framework for designing lensless imaging systems that use ultrafast detectors. We provide an algorithm for ideal sensor placement and an algorithm for optimized active illumination patterns. We show that efficient lensless imaging is possible with ultrafast measurement and compressive sensing. This paves the way for novel imaging architectures and remote sensing in extreme situations where imaging with a lens is not possible.
- 20 March 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 3 (3) , 398-407
Lensless Imaging With Compressive Ultrafast Sensing
Lensless imaging is an important and challenging problem. One notable solution to lensless imaging is a single-pixel camera that benefits from ideas central to compressive sampling. However, traditional single-pixel cameras require many illumination patterns that result in a long acquisition process. Here, we present a method for lensless imaging based on compressive ultrafast sensing. Each sensor acquisition is encoded with a different illumination pattern and produces a time series where time is a function of the photon's origin in the scene. Currently available hardware with picosecond time resolution enables time tagging photons as they arrive to an omnidirectional sensor. This allows lensless imaging with significantly fewer patterns compared to regular single-pixel imaging. To that end, we develop a framework for designing lensless imaging systems that use ultrafast detectors. We provide an algorithm for ideal sensor placement and an algorithm for optimized active illumination patterns. We show that efficient lensless imaging is possible with ultrafast measurement and compressive sensing. This paves the way for novel imaging architectures and remote sensing in extreme situations where imaging with a lens is not possible.