Thanh Nguyen

Reconstructing 3D geometric representation of objects with deep learning frameworks has recently gained a great deal of interest in numerous fields. The existing deep-learning-based 3D shape reconstruction techniques generally use a single red–green–blue (RGB) image, and the depth reconstruction accuracy is often highly limited due to a variety of reasons. We present a 3D shape reconstruction technique with an accuracy enhancement strategy by integrating the structured-light scheme with deep convolutional neural networks (CNNs). The key idea is to transform multiple (typically two) grayscale images consisting of fringe and/or speckle patterns into a 3D depth map using an end-to-end artificial neural network. Distinct from the existing autoencoder-based networks, the proposed technique reconstructs the 3D shape of target using a refinement approach that fuses multiple feature maps to obtain multiple outputs with an accuracy-enhanced final output. A few experiments have been conducted to verify the robustness and capabilities of the proposed technique. The findings suggest that the proposed network approach can be a promising 3D reconstruction technique for future academic research and industrial applications.

In recent years, research efforts in the field of digital holography have expanded significantly, due to the ability to obtain high-resolution intensity and phase images. The information contained in these images have become of great interest to the machine learning community, with applications spanning a wide portfolio of research areas, including bioengineering. In this work, we seek to demonstrate a high-fidelity simulation of holographic recording. By accurately and numerically simulating the propagation of a coherent light source through a series of optical elements and the object itself, we accurately predict the optical interference of the object and reference wave at the recording plane, including diffraction effects, aberrations, and speckle. We show that the optical transformation that predicts the complex field at the recording plane can be generalized for arbitrary holographic recording configurations using a matrix method. In addition, we provide a detailed description of digital phase reconstruction and aberration compensation for a variety of off-axis holographic configurations. Reconstruction errors are presented for the various holographic recording geometries and complex field objects. While the primary objective of this work is not to evaluate phase reconstruction approaches, the reconstruction of simulated holograms provides validation of the generalized simulation method. The long-term goal of this work is that the generalized holographic simulation motivates the use of phase reconstruction of the simulated holograms to populate databases for training machine-learning algorithms aimed at classifying relevant objects recorded through a variety of holographic setups.

Microscopy techniques, particularly the family of epifluorescence modalities, are workhorses of modern cell and molecular biology that enable microscale spatial insight. Intensity- and/or phase-based microscopy techniques such as brightfield, phase contrast, differential interference contrast, digital holography, Fourier ptychography, and optical diffraction tomography,^{1 – 7} among other modalities, have the potential to visualize the subcellular structure. However, these methods depend largely on light scattering that is defined by the internal structure-based index of refraction, which lacks biomolecular specificity. Fluorescence-based techniques, on the other hand, excite fluorophores, which act as labels to spatially localize biological molecules and structures within cells. These imaging techniques, especially fluorescence approaches, involve time-consuming preparation steps and costly reagents, introduce the possibility of signal bias due to photobleaching, and in time-lapse vital imaging, allow for possible misinterpretation of cell behavior due to gradual accumulation of sublethal damage from intense ultraviolet and other wavelengths used to excite molecular labels.⁸ Thus microscopy of cells is challenging due to the inherent trade-offs in sample preservation, image quality, and data acquisition time and the variability between labeling experiments.

Quantitative phase imaging (QPI) is a label-free optical imaging technique that measures the phase delay introduced when a coherent laser beam travels through a thin transparent specimen, such as cells.¹ The uncorrected optical pathlength yields information about cell morphology and geometric thickness of the specimen along its phase projection, as well as fluctuations in the local index of refraction.² Reconstructed optical phase maps that result from QPI of cells possess phase texture in the variation of pixel values within the boundary of the cell,³ quantified through texture parameters.⁴ The pixel information from phase maps is relevant to cell structure and function. For example, a cell’s refractive index can reveal its total protein concentration,⁵ organization, and distribution of subcellular organelles, which are distinctive for certain biological phenotypes.⁶ Hence, a collection of quantitative parameters from optical phase maps constitutes a cell type’s phase signature, providing additional information beyond the cell shape.

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequences of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by these large spatial ensembles so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800×10800 pixel phase image using only ∼25 seconds, a 50× speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ∼ 6×. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. We further propose a mixed loss function that combines the standard image domain loss and a weighted Fourier domain loss, which leads to improved reconstruction of the high frequency information. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution.

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequence of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by this large spatial ensemble so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800X10800 pixels phase image using only ~25 seconds, a 50X speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ~6X. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution.

Deep convolutional neural networks (DCNNs) offer a promising performance for many image processing areas, such as super-resolution, deconvolution, image classification, denoising, and segmentation, with outstanding results. Here, we develop for the first time, to our knowledge, a method to perform 3-D computational optical tomography using 3-D DCNN. A simulated 3-D phantom dataset was first constructed and converted to a dataset of phase objects imaged on a spatial light modulator. For each phase image in the dataset, the corresponding diffracted intensity image was experimentally recorded on a CCD. We then experimentally demonstrate the ability of the developed 3-D DCNN algorithm to solve the inverse problem by reconstructing the 3-D index of refraction distributions of test phantoms from the dataset from their corresponding diffraction patterns.

The noninvasive, fast acquisition of quantitative phase maps using digital holographic microscopy (DHM) allows tracking of rapid cellular motility on transparent substrates. On two‐dimensional surfaces in vitro, MDA‐MB‐231 cancer cells assume several morphologies related to the mode of migration and substrate stiffness, relevant to mechanisms of cancer invasiveness in vivo. The quantitative phase information from DHM may accurately classify adhesive cancer cell subpopulations with clinical relevance. To test this, cells from the invasive breast cancer MDA‐MB‐231 cell line were cultured on glass, tissue‐culture treated polystyrene, and collagen hydrogels, and imaged with DHM followed by epifluorescence microscopy after staining F‐actin and nuclei. Trends in cell phase parameters were tracked on the different substrates, during cell division, and during matrix adhesion, relating them to F‐actin features. Support vector machine learning algorithms were trained and tested using parameters from holographic phase reconstructions and cell geometric features from conventional phase images, and used to distinguish between elongated and rounded cell morphologies. DHM was able to distinguish between elongated and rounded morphologies of MDA‐MB‐231 cells with 94% accuracy, compared to 83% accuracy using cell geometric features from conventional brightfield microscopy. This finding indicates the potential of DHM to detect and monitor cancer cell morphologies relevant to cell cycle phase status, substrate adhesion, and motility. © 2017 International Society for Advancement of Cytometry

Quantitative 3D phase retrieval techniques are based on either interferometric techniques such as holography or noninterferometric intensity-based techniques such as the transport of intensity equation (TIE). Interferometric techniques are vibration-sensitive and often use a reference beam requiring complicated optical alignment. In this work we develop a simple, fast, and noninterferometric tomographic 3D phase retrieval technique based on the TIE which does not suffer from such drawbacks. The optical setup is a modified 4f TIE system which uses an SLM to replace the slow translation of the CCD required to record several diffraction patterns in a traditional TIE system. This novel TIE setup is suitable for dynamical events such as imaging biological processes. A rotating mechanical stage is constructed to obtain tomographic phase images of the object. The tomographic reconstruction algorithm is based on the Fourier slice theorem (backprojection algorithm) which applies to objects with a small refractive index span. Simulation and experimental results are shown as part of this work. A graphical user interface is developed to perform the TIE tomographic reconstruction algorithm and to synchronize the captured intensities by the CCD, the phase patterns displayed on the SLM, and the Arduino controlled rotating stage assembly.