Yong Li

This paper proposes a method composed of a loss function and a feature extractor structure, to learn the distinctive feature representation (DIFR) for descriptors on multimodal images. The distinctiveness of the feature representation plays a key role in matching keypoints, and has drawn a lot of research attention. In the past, many methods were proposed that either employed handcrafted features or deep neural networks to learn the feature representation. However, the incorrect ratio of built keypoint matches still remains high due to the insufficient distinctiveness of the associated descriptors. To address this problem, this work designs a new contrastive loss function that for the first time incorporates multiple negative descriptors from both the reference image and the test image, to constrain the networks to learn the distinctive feature representation for descriptors. To further enable the descriptors to encode more global information, we design RestExtractor, a novel CNN-Transformer structure that effectively adapts the self-attention mechanism to the feature learning for multimodal images. In the RestExtractor, we design a gated-v block and a receptive field enhanced feed-forward network for extracting distinctive modal-shared feature representation. We evaluated the proposed method on two challenging multimodal image datasets, VIS-SAR and VIS-IR. The experimental results showed that the proposed DIFR significantly improved the number of correct keypoint matches over the state-of-the-art methods. Our source code will be released on https://github.com/bohanlee/DIFR/

Feature representation plays a key role in matching keypoints, especially for the multispectral images of large spectral difference. On such image pairs, existing methods typically use the two images only, but it is challenging to directly learn spectrum invariant feature representation due to the complex non-linear distortion between them. To address this issue, this paper proposes using intermediate-band images to facilitate learning spectrum invariant feature representation. For this purpose, this work designs a spectrum adaptive gate network (SAG-Net) that consists of a spectrum gate (SPeG) module and a deep feature extractor. The SPeG module selectively activates the spectrum invariant features according to input image content on-the-fly. It hence allows for training on the images of over two bands simultaneously with a single network without the need of an individual branch per band. In order to investigate the SPeG module, we also constructed a L9-MSI dataset including 3167 scenes of aligned images across five spectral bands (visible, B5, B6, B7, and B10) from the Landsat 9 imagery. The experimental results demonstrate the SPeG module can learn the common feature representation for varying-band images, and the intermediate B5, B6, and B7 images are useful for the SAG-Net to learn the common feature between the visible and B10. On the L9-MSI dataset, the SAG-Net significantly improved the number of correct matches and the matching score. Our dataset will be released at https://github.com/bohanlee/L9MSI-Dataset.git.

In this paper, propose a fitness assistance algorithm based on the combination of a lightweight network human posture detection model, BlazePose, and a k-NN classification algorithm for real-time counting on mobile devices. Firstly, the BlazePose model is used to predict the target pose on the given sample image, and the obtained Landmark is normalized to build the training set of the k-NN classifier. Then Pose Classification Colab classifies each sample by targeting the entire training set while excluding outliers and underrepresented categories. Finally, the implementation of the classifier itself can realize the real-time counting of fitness exercises. In this study, squats and push-ups, two of the most representative sports, are selected as the verification objects. The experimental results show that, with real-time and accuracy guaranteed This algorithm is capable of classifying and counting push-ups and squats at a rate of 30 frames on an ordinary mobile device.