Jun Yu

Motion deblurring in scenes involving small moving objects or low-illumination conditions is challenging. This paper presents an effective deep-learning solution that utilizes correlation images as key auxiliaries to address the problem. The correlation image, produced by a three-phase correlation image sensor (3PCIS), is a temporal correlation between incident light and reference signals within a frame time, which encodes intensity changes of incident light over the exposure time. Since correlation images explicitly record motion information lost during the blurring process during exposure, they can be used for accurately identifying the location and degree of blur, especially in low-illumination conditions and scenarios with small moving objects. Therefore, we combine correlation images and motion-blurred images as inputs and build a two-stream network for motion deblurring. Two key designs in our model are 1) Shared-gated Block (SGB), which enables information exchange between the two encoders and selectively allows useful information to pass through the network to obtain high-quality output; 2) a Motion-guided Block (MGB), decoding process that can draw more attention to the blurred areas in the image, thereby achieving clearer textures and details restoration in the blurred areas. The experimental results show that our model not only can successfully eliminate the motion blur in the above challenging scenarios, but also achieves a state-of-the-art 36.02dB in Peak Signal-to-Noise Ratio (PSNR) on the GoPro dataset with simulated correlation images.

Single-frame optical flow estimation is a more challenging task than predicting the optical flow between adjacent frames in a video. This paper presents a two-stream network that combines motion information from correlation images with appearance information from intensity images to estimate optical flow in a single-frame manner. The correlation image generated by a three-phase correlation image sensor (3PCIS) records changes in incident intensity during the exposure time and conveys motion information about moving objects. This is crucial to assist motion-blurred images in estimating the motion state of moving objects. Due to the lack of directly available datasets to train our network, we generate a synthetic dataset. Importantly, we introduce a definition that describes optical flow on a single motion-blurred image, which is essential for creating reasonable ground truth. Our experimental results demonstrate that 1) network trained on our synthetic dataset achieves an average End Point Error (EPE) of 0.357 and generalizes well to real-world scenes; 2) our proposed single-frame method outperforms conventional two-frame-based methods in motion-blurred scenes.

In cognitive science, the bottleneck of information processing capabilities promotes humans to selectively focus on part of the visible information while ignoring the rest. This is usually called the attention mechanism. Among different attention mechanisms, spectrum attention focuses on assigning every frequency channel obtained by DCT an appropriate weight to achieve adaptive filtering and sends the reweighting frequency components into a real-valued neural network. However, DCT is a real-valued transformation for signal processing thus phase information tends to be omitted. Here we consider a departure from these conventions and propose a spectrum attention complex neural network assisted by DFT, to make a holistic frequency analysis in a completely complex domain. Our proposed network demonstrates outstanding performance on a one-dimensional hyperspectral dataset and MNIST dataset, compared to both traditional complex neural networks and DCT based frequency attention methods.

It seems difficult to recognize an object from its background with similar color using conventional segmentation methods. An efficient way is to utilize hyperspectral images that contain more wave bands and richer information than only RGB components. Particularly in our task, we aim to separate a pepper from densely packed green leaves for automatic picking in agriculture. Given that hyperspectral imaging can be regarded as a kind of wave propagation process, we make a novel attempt of introducing a complex neural network tailored for wave-related problems. Due to the lack of hyperspectral data, pixelwise training is deployed, and 1D fast Fourier transform of the hyperspectral data is used for the construction of complex input. Experimental results have showcased that a complex neural network outperforms a real-valued one in terms of detection accuracy by 3.9% and F1 score by 1.33%. Moreover, it enables the ability to select frequency bands used such as low-frequency components to boost performance as well as prevent overfitting problems for learning more generalization features. Thus, we put forward a lightweight pixelwise complex model for hyperspectral-related problems and provide an efficient way for green pepper automatic picking in agriculture using small datasets.

There is a growing demand for developing image sensor systems to aid fruit and vegetable harvesting, and crop growth prediction in precision agriculture. In this paper, we present an end-to-end optimization approach for the simultaneous design of optical filters and green pepper segmentation neural networks. Our optimization method modeled the optical filter as one learnable neural network layer and attached it to the subsequent camera spectral response (CSR) layer and segmentation neural network for green pepper segmentation. We used not only the standard red–green–blue output from the CSR layer but also the color-ratio maps as additional cues in the visible wavelength and to augment the feature maps as the input for segmentation. We evaluated how well our proposed color-ratio maps enhanced optical filter design methods in our collected dataset. We find that our proposed method can yield a better performance than both an optical filter RGB system without color-ratio maps and a raw RGB camera (without an optical filter) system. The proposed learning-based framework can potentially build better image sensor systems for green pepper segmentation.

Detecting and identifying objects of similar color is a challenging task in computer vision. Green peppers in a natural environment can be found using the abundant information provided by a hyperspectral camera in the spectral domain, but the hyperspectral camera is an expensive device. Therefore, we propose a novel framework called Optical Filter Net, which enables the design of an optical filter that improves the performance of green pepper segmentation by a specific red-green-blue (RGB) camera system. When installed with the optical filter, the system can efficiently utilize the spectral information in the visible wavelength to distinguish green pepper and foliage without requiring an expensive hyperspectral camera. A main finding is the similarity between the transmission curve of the optical filter and the depth-wise convolution kernel without bias. Accordingly, we can treat the transmission curve of the optical filter as one layer of a deep neural network. The whole structure can be trained in an end-to-end manner. To comply with the physical requirement of the optical filter, we further constrain the training process to achieve a non-negative and smooth transmission curve. In an experimental evaluation on our dataset, our proposed spectral-aware RGB camera system outperformed the RGB camera system without an optical filter.