MTMatthew Tancik

Matthew Tancik

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Matthew Tancik has a total of 25 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers

    GARField: Group Anything with Radiance Fields

    by Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, Angjoo Kanazawa

    Grouping is inherently ambiguous due to the multiple levels of granularity in which one can decompose a sceneshould the wheels of an excavator be considered separate or part of the whole? We propose Group Anything with Radiance Fields (GARField), an approach for decomposing 3D scenes into a hierarchy of semantically meaningful groups from posed image inputs. To do this we embrace group ambiguity through physical scale: by optimizing a scaleconditioned 3D affinity feature field, a point in the world can belong to different groups of different sizes. We optimize this field from a set of 2D masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine hierarchy, using scale to consistently fuse conflicting masks from different viewpoints. From this field we can derive a hierarchy of possible groupings via automatic tree construction or user interaction. We evaluate GARField on a variety of in-the-wild scenes and find it effectively extracts groups at many levels: clusters of objects, objects, and various subparts. GARField inherently represents multi-view consistent groupings and produces higher fidelity groups than the input SAM masks. GARField's hierarchical grouping could have exciting downstream applications such as 3D asset extraction or dynamic scene understanding. Project site: https://www.garfield.studio/


  • Top Cited Papers
    Open Access

    Nerfstudio: A Modular Framework for Neural Radiance Field Development

    by Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al.


  • Open Access

    The One Where They Reconstructed 3D Humans and Environments in TV Shows

    by Georgios Pavlakos, Ethan Weber, Matthew Tancik, Angjoo Kanazawa

    TV shows depict a wide variety of human behaviors and have been studied extensively for their potential to be a rich source of data for many applications. However, the majority of the existing work focuses on 2D recognition tasks. In this paper, we make the observation that there is a certain persistence in TV shows, i.e., repetition of the environments and the humans, which makes possible the 3D reconstruction of this content. Building on this insight, we propose an automatic approach that operates on an entire season of a TV show and aggregates information in 3D; we build a 3D model of the environment, compute camera information, static 3D scene structure and body scale information. Then, we demonstrate how this information acts as rich 3D context that can guide and improve the recovery of 3D human pose and position in these environments. Moreover, we show that reasoning about humans and their environment in 3D enables a broad range of downstream applications: re-identification, gaze estimation, cinematography and image editing. We apply our approach on environments from seven iconic TV shows and perform an extensive evaluation of the proposed system.

  • Photography optics in the time dimension

    by Barmak Heshmat, Matthew Tancik, Guy Satat, Ramesh Raskar


  • Top Cited Papers

    Towards photography through realistic fog

    by Guy Satat, Matthew Tancik, Ramesh Raskar

    Imaging through fog has important applications in industries such as self-driving cars, augmented driving, airplanes, helicopters, drones and trains. Here we show that time profiles of light reflected from fog have a distribution (Gamma) that is different from light reflected from objects occluded by fog (Gaussian). This helps to distinguish between background photons reflected from the fog and signal photons reflected from the occluded object. Based on this observation, we recover reflectance and depth of a scene obstructed by dense, dynamic, and heterogeneous fog. For practical use cases, the imaging system is designed in optical reflection mode with minimal footprint and is based on LIDAR hardware. Specifically, we use a single photon avalanche diode (SPAD) camera that time-tags individual detected photons. A probabilistic computational framework is developed to estimate the fog properties from the measurement itself, without prior knowledge. Other solutions are based on radar that suffers from poor resolution (due to the long wavelength), or on time gating that suffers from low signal-to-noise ratio. The suggested technique is experimentally evaluated in a wide range of fog densities created in a fog chamber It demonstrates recovering objects 57cm away from the camera when the visibility is 37cm. In that case it recovers depth with a resolution of 5cm and scene reflectance with an improvement of 4dB in PSNR and 3.4× reconstruction quality in SSIM over time gating techniques.


  • Data-Driven Non-Line-of-Sight Imaging With A Traditional Camera

    by Matthew Tancik, Tristan Swedish, Guy Satat, Ramesh Raskar

    Non-line-of-right (NLOS) imaging has recently been demonstrated using traditional cameras. Here we present a data-driven technique for NLOS imaging. We experimentally analyze the role of scene geometry and clutter on reconstruction quality.


  • Introduction to Time-folded Optics; Spatial Compression of Paraxial Optics Using Time

    by Barmak Heshmat, Matthew Tancik, Guy Satat, Ramesh Raskar

    We exploit the high time resolution of ultrafast imaging sensors to fold and modify the imaging optics of the camera. We demonstrate an order of magnitude compression in lens–sensor distance using a time-folded design.

  • Top Cited Papers
    Open Access

    Object classification through scattering media with deep learning on time resolved measurement

    by Guy Satat, Matthew Tancik, Otkrist Gupta, Barmak Heshmat, Ramesh Raskar

    We demonstrate an imaging technique that allows identification and classification of objects hidden behind scattering media and is invariant to changes in calibration parameters within a training range. Traditional techniques to image through scattering solve an inverse problem and are limited by the need to tune a forward model with multiple calibration parameters (like camera field of view, illumination position etc.). Instead of tuning a forward model and directly inverting the optical scattering, we use a data driven approach and leverage convolutional neural networks (CNN) to learn a model that is invariant to calibration parameters variations within the training range and nearly invariant beyond that. This effectively allows robust imaging through scattering conditions that is not sensitive to calibration. The CNN is trained with a large synthetic dataset generated with a Monte Carlo (MC) model that contains random realizations of major calibration parameters. The method is evaluated with a time-resolved camera and multiple experimental results are provided including pose estimation of a mannequin hidden behind a paper sheet with 23 correct classifications out of 30 tests in three poses (76.6% accuracy on real-world measurements). This approach paves the way towards real-time practical non line of sight (NLOS) imaging applications.


  • Lensless Imaging With Compressive Ultrafast Sensing

    by Guy Satat, Matthew Tancik, Ramesh Raskar

    Lensless imaging is an important and challenging problem. One notable solution to lensless imaging is a single-pixel camera that benefits from ideas central to compressive sampling. However, traditional single-pixel cameras require many illumination patterns that result in a long acquisition process. Here, we present a method for lensless imaging based on compressive ultrafast sensing. Each sensor acquisition is encoded with a different illumination pattern and produces a time series where time is a function of the photon's origin in the scene. Currently available hardware with picosecond time resolution enables time tagging photons as they arrive to an omnidirectional sensor. This allows lensless imaging with significantly fewer patterns compared to regular single-pixel imaging. To that end, we develop a framework for designing lensless imaging systems that use ultrafast detectors. We provide an algorithm for ideal sensor placement and an algorithm for optimized active illumination patterns. We show that efficient lensless imaging is possible with ultrafast measurement and compressive sensing. This paves the way for novel imaging architectures and remote sensing in extreme situations where imaging with a lens is not possible.