Visible-to-Thermal Transfer Learning for Facial Landmark Detection
Open Access
- 31 March 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 52759-52772
Abstract
There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.Keywords
This publication has 32 references indexed in Scilit:
- Deep Learning Face Representation from Predicting 10,000 Classes2014
- DeepFace: Closing the Gap to Human-Level Performance in Face Verification2014
- Eye localization from thermal infrared imagesPattern Recognition, 2013
- Eye detection in the Middle-Wave Infrared spectrum: Towards recognition in the dark2011
- On Person Authentication by Fusing Visual and Thermal Face Biometrics2006
- Face recognition by fusing thermal infrared and visible imageryImage and Vision Computing, 2006
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005
- Thermal face recognition in an operational scenario2004
- SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK1994
- On Information and SufficiencyThe Annals of Mathematical Statistics, 1951