Shay Elmalem

Temporal photon correlations have been a crucial resource for quantum and quantum-enabled optical science for over half a century. However, attaining non-classical information through these correlations has typically been limited to a single point (or, at best, a few points) at a time. Here, we perform a massively multiplexed wide-field photon correlation measurement using a large single-photon avalanche diode array, the SwissSPAD3. We demonstrate the performance of this apparatus by acquiring wide-field photon correlation measurements of single-photon emitters and illustrate two applications of the attained quantum information: wide-field emitter counting and quantum-enabled super-resolution imaging (by a factor of 500×500). The considerations and limitations of applying this technique in a practical context are discussed. Ultimately, the realization of massively multiplexed wide-field photon correlation measurements can accelerate quantum sensing protocols and quantum-enabled imaging techniques by orders of magnitude.

Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance the capabilities of existing cameras. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely-digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works proposed to address these limitations using non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations with simpler means, we propose a hybrid optical-digital method for video reconstruction that requires only simple modifications to existing optical systems. We use a learned dynamic phase-coding in the lens aperture during the image acquisition to encode the motion trajectories, which serve as prior information for the video reconstruction process. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image, using an image-to-video convolutional neural network. We present advantages and improved performance compared to existing methods, using both simulations and a real-world camera prototype. We extend our optical coding also to video frame interpolation and present robust and improved results for noisy videos.

Stereo imaging is the most common passive method for producing reliable depth maps. Calibration is a crucial step for every stereo-based system, and despite all the advancements in the field, most calibrations are still done by the same tedious method using a checkerboard target. Monocular-based depth estimation methods do not require extrinsic calibration but generally achieve inferior depth accuracy. In this paper, we present a novel online self-calibration method, which makes use of both stereo and monocular depth maps to find the transformation required for extrinsic calibration by enforcing consistency between both maps. The proposed method works in a closed-loop and exploits the pre-trained networks' global context, and thus avoids feature matching and outliers issues. In addition to presenting our method using an image-based monocular depth estimation method, which can be implemented in most systems without additional changes, we also show that adding a phase-coded aperture mask leads to even better and faster convergence. We demonstrate our method on road scenes from the KITTI vision benchmark and real-world scenes using our prototype camera. Our code is publicly available at https://github.com/YotYot/CalibrationNet.

Polarization imaging generally requires either a designated sensor or a sequence of polarization-filtered images. We propose a lensless polarization camera, based on a conventional sensor, diffuser, simple polarization filter, and restoration algorithm.

Motion-related image blur is a known issue in photography. In practice, it limits the exposure time while capturing moving objects; thus, achieving proper exposure is difficult. Extensive research has been carried out to compensate for it, to allow increased light throughput without motion artifacts. In this work, a joint optical-digital processing method for motion deblurring is proposed and demonstrated. Using dynamic phase coding in the lens aperture during the image acquisition, the motion trajectory is encoded in an intermediate optical image. This coding embeds cues for both the motion direction and extent by coloring the spatial blur of each object. These color cues serve as guidance for a digital deblurring process, implemented using a convolutional neural network (CNN) trained to utilize such coding for image restoration. Particularly, unlike previous optical coding solutions, our strategy encodes cues with no limitation on the motion direction, and without sacrificing light efficiency. We demonstrate the advantage of the proposed approach over blind deblurring methods with no optical coding, as well as over other solutions that use coded acquisition, in both simulation and real-world experiments.

Motion deblurring solution based on spatio-temporal coding is proposed. Using aperture coding and focus variations during exposure, a joint spatio-temporal coding is achieved, which is in-turn utilized for motion deblurring in the post processing step.

Depth estimation from a single image is a well known challenge in computer vision. With the advent of deep learning, several approaches for monocular depth estimation have been proposed, all of which have inherent limitations due to the scarce depth cues that exist in a single image. Moreover, these methods are very demanding computationally, which makes them inadequate for systems with limited processing power. In this paper, a phase-coded aperture camera for depth estimation is proposed. The camera is equipped with an optical phase mask that provides unambiguous depth-related color characteristics for the captured image. These are used for estimating the scene depth map using a fully-convolutional neural network. The phase-coded aperture structure is learned jointly with the network weights using back-propagation. The strong depth cues (encoded in the image by the phase mask, designed together with the network weights) allow a much simpler neural network architecture for faster and more accurate depth estimation. Performance achieved on simulated images as well as on a real optical setup is superior to state-of-the-art monocular depth estimation methods (both with respect to the depth accuracy and required processing power), and is competitive with more complex and expensive depth estimation methods such as light-field cameras.

Modern consumer electronics market dictates the need for small-scale and high-performance cameras. Such designs involve trade-offs between various system parameters. In such trade-offs, Depth Of Field (DOF) is a significant issue very often. We propose a computational imaging-based technique to overcome DOF limitations. Our approach is based on the synergy between a simple phase aperture coding element and a convolutional neural network (CNN). The phase element, designed for DOF extension using color diversity in the imaging system response, causes chromatic variations by creating a different defocus blur for each color channel of the image. The phase-mask is designed such that the CNN model is able to restore from the coded image an all-in-focus image easily. This is achieved by using a joint end-to-end training of both the phase element and the CNN parameters using backpropagation. The proposed approach provides superior performance to other methods in simulations as well as in real-world scenes.

Single image depth estimation is achieved using computational imaging and Deep Learning (DL). Imaging with phase-mask is also modeled as a DL-layer, and the mask and DL parameters are jointly designed using labeled data.