Mengyang Liu

We propose a hybrid recurrent Video Colorization with Hybrid Generative Adversarial Network (VCGAN), an improved approach to video colorization using end-to-end learning. The VCGAN addresses two prevalent issues in the video colorization domain: Temporal consistency and unification of colorization network and refinement network into a single architecture. To enhance colorization quality and spatiotemporal consistency, the mainstream of generator in VCGAN is assisted by two additional networks, i.e., global feature extractor and placeholder feature extractor, respectively. The global feature extractor encodes the global semantics of grayscale input to enhance colorization quality, whereas the placeholder feature extractor acts as a feedback connection to encode the semantics of the previous colorized frame in order to maintain spatiotemporal consistency. If changing the input for placeholder feature extractor as grayscale input, the hybrid VCGAN also has the potential to perform image colorization. To improve the consistency of far frames, we propose a dense long-term loss that smooths the temporal disparity of every two remote frames. Trained with colorization and temporal losses jointly, VCGAN strikes a good balance between color vividness and video continuity. Experimental results demonstrate that VCGAN produces higher-quality and temporally more consistent colorful videos than existing approaches.

Deep Learning based image quality assessment (IQA) has been shown to greatly improve the quality score prediction accuracy of images with single distortion. However, because these models lack generalizability and the accuracy of multidistortion-based image data is relatively low, designing reliable IQA systems is still an open issue. In this paper, we propose to introduce long-range dependencies between local artifacts and high-order spatial pooling into a convolutional neural network (CNN) model to improve the performance and generalizability of the full-reference IQA (FR-IQA). This long-range dependencies model is based on the hypothesis that local apparent artifacts can affect the overall image quality. The proposed network architecture adopts a non-local means algorithm to establish connections between all positions in the deep feature space and uses the Minkowski function to improve the non-linearity of the spatial pooling. Based on this architecture, a robust FR-IQA system has been constructed and evaluated on three well-known single-distortion-based IQA databases (LIVE, CSIQ, and TID2013) and a multidistortion-based IQA database (MDID). Experimental results demonstrate that, compared with the latest FR-IQA systems, the proposed long-range dependencies-boosted CNN-based FR-IQA system can achieve state-of-the-art performance. A comprehensive cross-database evaluation also shows that the proposed system is sufficiently generalized between different databases and multidistortion-based image data is more useful for training robust image quality metrics.

Deep learning based image hashing methods learn hash codes by using powerful feature extractors and nonlinear transformations to achieve highly efficient image retrieval. For most end-to-end deep hashing methods, the supervised learning process relies on pair-wise or triplet-wise information to provide an internal relationship of similarity data. However, the use of pair-wise and triplet loss function is limited not only by expensive training costs but also by quantization errors. In this paper, we propose a novel semantic learning based hashing method for image retrieval to optimize the deep features structure in the hash space from a perspective of angular view. Specifically, we proposed an angular hashing loss function that explicitly improve intra-class compactness and inter-class separability between features in hash space. Geometrically, angular hashing loss can be regarded as imposing hash constraints on hypersphere manifold. In order to solve the training problem on the multi-label case, we further designed a dynamic Softmax training strategy that can directly train the network using gradient descent method. Extensive experiments on two well-known datasets of CIFAR-10 and NUS-WIDE demonstrate that the proposed Angular Deep Supervised Hashing (ADSH) method can generate high-quality and compact binary codes, which can achieve state-of-the-art performance as compared with conventional image hashing and deep learning-based hashing methods.

Face recognition based-access systems have been used widely in security systems as the recognition accuracy can be quite high. However, these systems suffer from low robustness to spoofing attacks. To achieve a reliable security system, a well-defined face liveness detection technique is crucial. We present an approach for this problem by combining data of the light-field camera (LFC) and the convolutional neural networks in the detection process. The LFC can detect the depth of an object by a single shot, from which we derive meaningful features to distinguish the spoofing attack from the real face, through a single shot. We propose two features for liveness detection: the ray difference images and the microlens images. Experimental results based on a self-built light-field imaging database for three types of the spoofing attacks are presented. The experimental results show that the proposed system gives a lower average classification error (0.028) as compared with the method of using hand-crafted features and conventional imaging systems. In addition, the proposed system can be used to classify the type of the spoofing attack.

Deep Convolutional Neural Networks (CNNs) have been successfully applied on no-reference image quality assessment (NR-IQA) with respect to human perception. Most of these methods deal with small image patches and use the average score of the test patches for predicting the whole image quality. We discovered that image patches from homogenous regions are unreliable for both neural network training and final image quality score estimation. In addition, image patches with complex structures have much higher chances to achieve better image quality prediction. Based on these findings, we enhanced the conventional CNN-based NR-IQA algorithm to avoid homogenous patches for the network training and quality score estimation. Moreover, we also use a variance-based weighting average to bias the final image quality score to the patches with complex structure. Experimental results show that this simple approach can achieve state-of-the-art performance as compared with well-known NR-IQA algorithms.