Benjamin S. Riggan
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Benjamin S. Riggan has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
Shuowen Hu
U.S. Army Combat Capabilities Development Command, Army Research Laboratory, Adelphi, MD, USA
3 shared publications- Cedric Nimpa Fondje
University of NebraskaLincoln, Lincoln, NE, USA
2 shared publications - Kshitij Nikhal
University of NebraskaLincoln, Lincoln, NE, USA
2 shared publications - Nasser M. Nasrabadi
West Virginia University
2 shared publications - Bangalore S. Manjunath
University of California, Santa Barbara, California, United States of America
1 shared publication Chi-En Chiang
1 shared publication
Latest Publications
- Open Access
- 13 August 2026
- Published by Elsevier in SSRN Electronic Journal
WeedExpert-R1: Incentivizing Botanical Reasoning in MLLMs with Reinforcement Learning for Precision Weed Grounding
Precision weed control requires both species-level identification and per-instance localization, butconventional object detectors are constrained by a closed-vocabulary paradigm that hinders cross-regiondeployment and lack the reasoning capability to justify their predictions in complex agricultural scenes.Recently, multimodal large language models (MLLMs) have demonstrated strong visual perceptionand reasoning in visual grounding, offering a promising alternative. However, their insufficient domainspecificbotanical knowledge often leads to hallucinations during fine-grained weed identification. Inthis study,WeedExpert-R1 is introduced, a multimodal reasoning model that, following the R1-styletraining paradigm, incentivizes visually grounded botanical reasoning through verifiable rewards.Specifically, a domain-specific Chain-of-Thought (CoT) synthesis pipeline is proposed. It pairs ahuman-curated botanical trait dictionary, covering leaf shape, margin, petiole, and stem morphology,with an Auditor–Synthesizer LLM workflow to generate high-quality reasoning data for supervisedfine-tuning as a cold start. This pipeline bridges the gap between textual botanical knowledge and thevisual perception of MLLMs. Group Relative Policy Optimization (GRPO) is then applied withverifiable rewards (format, accuracy, count, and length penalty) to further enhance the model’scapability to perceive and localize diverse weeds. WeedExpert-R1-4B achieves 75.82%, 89.30%,and 87.81% on Precision@(F1=1, IoU≥0.5), Precision@0.5, and Recall@0.5 across 37 weed specieson a benchmark suite of six weed datasets (3SeasonWeedDet10, CottonWeedDet12, CottonWeedDet3,Weed-crop,Weed25, and PREEC), and outperforms both frontier proprietary models (GPT-5.4, Gemini-3.1-Pro) and larger open-source baselines (Qwen3-VL-30B-Instruct, Gemma-4-31B-it). Moreover,qualitative results on unseen species further illustrate the open-vocabulary capability ofWeedExpert-R1,suggesting potential for deployment across diverse regions and crops without retraining.
- 13 August 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessWeedExpert-R1: Incentivizing Botanical Reasoning in MLLMs with Reinforcement Learning for Precision Weed Grounding
Precision weed control requires both species-level identification and per-instance localization, butconventional object detectors are constrained by a closed-vocabulary paradigm that hinders cross-regiondeployment and lack the reasoning capability to justify their predictions in complex agricultural scenes.Recently, multimodal large language models (MLLMs) have demonstrated strong visual perceptionand reasoning in visual grounding, offering a promising alternative. However, their insufficient domainspecificbotanical knowledge often leads to hallucinations during fine-grained weed identification. Inthis study,WeedExpert-R1 is introduced, a multimodal reasoning model that, following the R1-styletraining paradigm, incentivizes visually grounded botanical reasoning through verifiable rewards.Specifically, a domain-specific Chain-of-Thought (CoT) synthesis pipeline is proposed. It pairs ahuman-curated botanical trait dictionary, covering leaf shape, margin, petiole, and stem morphology,with an Auditor–Synthesizer LLM workflow to generate high-quality reasoning data for supervisedfine-tuning as a cold start. This pipeline bridges the gap between textual botanical knowledge and thevisual perception of MLLMs. Group Relative Policy Optimization (GRPO) is then applied withverifiable rewards (format, accuracy, count, and length penalty) to further enhance the model’scapability to perceive and localize diverse weeds. WeedExpert-R1-4B achieves 75.82%, 89.30%,and 87.81% on Precision@(F1=1, IoU≥0.5), Precision@0.5, and Recall@0.5 across 37 weed specieson a benchmark suite of six weed datasets (3SeasonWeedDet10, CottonWeedDet12, CottonWeedDet3,Weed-crop,Weed25, and PREEC), and outperforms both frontier proprietary models (GPT-5.4, Gemini-3.1-Pro) and larger open-source baselines (Qwen3-VL-30B-Instruct, Gemma-4-31B-it). Moreover,qualitative results on unseen species further illustrate the open-vocabulary capability ofWeedExpert-R1,suggesting potential for deployment across diverse regions and crops without retraining.
- Open Access
- 22 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 148902-148916
Cross-Spectral Attention for Unsupervised RGB-IR Face Verification and Person Re-Identification
Cross-spectral biometrics, such as matching imagery of faces or persons from visible (RGB) and thermal infrared (IR) bands, have rapidly advanced over the last decade because of the increasing sensitivity, size, quality, and ubiquity of IR focal plane arrays and enhanced analytics beyond the visible spectrum. Current techniques for mitigating large spectral disparities between RGB and IR imagery often include learning a discriminative common subspace by exploiting precisely curated data acquired from multiple spectra. Although there are challenges with determining robust architectures for extracting common information, a critical limitation for supervised methods is poor scalability in terms of acquiring labeled data. Therefore, we propose a novel unsupervised cross-spectral framework that combines (1) a new pseudo triplet loss with cross-spectral voting, (2) a new cross-spectral attention network leveraging multiple subspaces, and (3) structured sparsity to perform more discriminative cross-spectral clustering.We extensively compare our proposed RGB-IR biometric learning framework (and its individual components) with recent and previous state-of-the-art models on two challenging benchmark datasets: DEVCOM Army Research Laboratory Visible-Thermal Face Dataset (ARL-VTF) and RegDB person re-identification dataset, and, in some cases, achieve performance superior to completely supervised methods.
- 22 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 148902-148916
Open AccessCross-Spectral Attention for Unsupervised RGB-IR Face Verification and Person Re-Identification
Cross-spectral biometrics, such as matching imagery of faces or persons from visible (RGB) and thermal infrared (IR) bands, have rapidly advanced over the last decade because of the increasing sensitivity, size, quality, and ubiquity of IR focal plane arrays and enhanced analytics beyond the visible spectrum. Current techniques for mitigating large spectral disparities between RGB and IR imagery often include learning a discriminative common subspace by exploiting precisely curated data acquired from multiple spectra. Although there are challenges with determining robust architectures for extracting common information, a critical limitation for supervised methods is poor scalability in terms of acquiring labeled data. Therefore, we propose a novel unsupervised cross-spectral framework that combines (1) a new pseudo triplet loss with cross-spectral voting, (2) a new cross-spectral attention network leveraging multiple subspaces, and (3) structured sparsity to perform more discriminative cross-spectral clustering.We extensively compare our proposed RGB-IR biometric learning framework (and its individual components) with recent and previous state-of-the-art models on two challenging benchmark datasets: DEVCOM Army Research Laboratory Visible-Thermal Face Dataset (ARL-VTF) and RegDB person re-identification dataset, and, in some cases, achieve performance superior to completely supervised methods.
- 9 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 5 (2) , 170-182
Multi-Context Grouped Attention for Unsupervised Person Re-Identification
Recent advancements like multiple contextual analysis, attention mechanisms, distance-aware optimization, and multi-task guidance have been widely used for supervised person re-identification (ReID), but the implementation and effects of such methods in unsupervised ReID frameworks are non-trivial and unclear, respectively. Moreover, with increasing size and complexity of image-and video-based ReID datasets, manual or semi-automated annotation procedures for supervised ReID are becoming labor intensive and cost prohibitive, which is undesirable especially considering the likelihood of annotation errors increase with scale/complexity of data collections. Therefore, we propose a new iterative clustering framework that is insensitive to annotation errors and over-fitting ReID annotations (i.e., labels). Our proposed unsupervised framework incorporates (a) a novel multi-context group attention architecture that learns a holistic attention map from multiple local and global contexts, (b) an unsupervised clustering loss function that down-weights easily discriminative identities, and (c) a background diversity term that helps cluster persons across different cross-camera views without leveraging any identification or camera labels. We perform extensive analysis using the DukeMTMC-VideoReID and MARS video-based ReID datasets and the MSMT17 image-based ReID dataset. Our approach is shown to provide a new state-of-the-art performance for unsupervised ReID, reducing the rank-1 performance gap between supervised and unsupervised ReID to 1.1%, 12.1%, and 21.9% from 6.1%, 17.9%, and 22.6% for DukeMTMC, MARS, and MSMT17 datasets, respectively.
- 9 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 5 (2) , 170-182
Multi-Context Grouped Attention for Unsupervised Person Re-Identification
Recent advancements like multiple contextual analysis, attention mechanisms, distance-aware optimization, and multi-task guidance have been widely used for supervised person re-identification (ReID), but the implementation and effects of such methods in unsupervised ReID frameworks are non-trivial and unclear, respectively. Moreover, with increasing size and complexity of image-and video-based ReID datasets, manual or semi-automated annotation procedures for supervised ReID are becoming labor intensive and cost prohibitive, which is undesirable especially considering the likelihood of annotation errors increase with scale/complexity of data collections. Therefore, we propose a new iterative clustering framework that is insensitive to annotation errors and over-fitting ReID annotations (i.e., labels). Our proposed unsupervised framework incorporates (a) a novel multi-context group attention architecture that learns a holistic attention map from multiple local and global contexts, (b) an unsupervised clustering loss function that down-weights easily discriminative identities, and (c) a background diversity term that helps cluster persons across different cross-camera views without leveraging any identification or camera labels. We perform extensive analysis using the DukeMTMC-VideoReID and MARS video-based ReID datasets and the MSMT17 image-based ReID dataset. Our approach is shown to provide a new state-of-the-art performance for unsupervised ReID, reducing the rank-1 performance gap between supervised and unsupervised ReID to 1.1%, 12.1%, and 21.9% from 6.1%, 17.9%, and 22.6% for DukeMTMC, MARS, and MSMT17 datasets, respectively.
- 21 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 5 (1) , 15-28
Learning Domain and Pose Invariance for Thermal-to-Visible Face Recognition
Interest in thermal to visible face recognition has grown significantly over the last decade due to advancements in thermal infrared cameras and analytics beyond the visible spectrum. Despite large discrepancies between thermal and visible spectra, existing approaches bridge domain gaps by either synthesizing visible faces from thermal faces or by learning the cross-spectrum image representations. These approaches typically work well with frontal facial imagery collected at varying ranges and expressions, but exhibit significantly reduced performance when matching thermal faces with varying poses to frontal visible faces. We propose a novel Domain and Pose Invariant Framework that simultaneously learns domain and pose invariant representations. Our proposed framework is composed of modified networks for extracting the most correlated intermediate representations from off-pose thermal and frontal visible face imagery, a sub-network to jointly bridge domain and pose gaps, and a joint-loss function comprised of cross-spectrum and pose-correction losses. We demonstrate efficacy and advantages of the proposed method by evaluating on three thermal-visible datasets: ARL Visible-to-Thermal Face, ARL Multimodal Face, and Tufts Face. Although DPIF focuses on learning to match off-pose thermal to frontal visible faces, we also show that DPIF enhances performance when matching frontal thermal face images to frontal visible face images.
- 21 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 5 (1) , 15-28
Learning Domain and Pose Invariance for Thermal-to-Visible Face Recognition
Interest in thermal to visible face recognition has grown significantly over the last decade due to advancements in thermal infrared cameras and analytics beyond the visible spectrum. Despite large discrepancies between thermal and visible spectra, existing approaches bridge domain gaps by either synthesizing visible faces from thermal faces or by learning the cross-spectrum image representations. These approaches typically work well with frontal facial imagery collected at varying ranges and expressions, but exhibit significantly reduced performance when matching thermal faces with varying poses to frontal visible faces. We propose a novel Domain and Pose Invariant Framework that simultaneously learns domain and pose invariant representations. Our proposed framework is composed of modified networks for extracting the most correlated intermediate representations from off-pose thermal and frontal visible face imagery, a sub-network to jointly bridge domain and pose gaps, and a joint-loss function comprised of cross-spectrum and pose-correction losses. We demonstrate efficacy and advantages of the proposed method by evaluating on three thermal-visible datasets: ARL Visible-to-Thermal Face, ARL Multimodal Face, and Tufts Face. Although DPIF focuses on learning to match off-pose thermal to frontal visible faces, we also show that DPIF enhances performance when matching frontal thermal face images to frontal visible face images.
- Open Access
- 31 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 52759-52772
Visible-to-Thermal Transfer Learning for Facial Landmark Detection
There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.
- 31 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 52759-52772
Open AccessVisible-to-Thermal Transfer Learning for Facial Landmark Detection
There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.
- 18 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 3 (2) , 266-280
Multi-Scale Thermal to Visible Face Verification via Attribute Guided Synthesis
Thermal-to-visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or learn domain-invariant robust features from these modalities for cross-modal matching. In this paper, we use attributes extracted from visible images to synthesize attribute-preserved visible images from thermal imagery for cross-modal matching. A pre-trained attribute predictor network is used to extract the attributes from the visible image. Then, a novel multi-scale generator is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. Finally, a pre-trained VGG-Face network is leveraged to extract features from the synthesized image and the input visible image for verification. Extensive experiments evaluated on three datasets (ARL Face Database, Visible and Thermal Paired Face Database, and Tufts Face Database) demonstrate that the proposed method achieves state-of-the-art performance. In particular, it achieve around 2.41%, 2.85% and 1.77% improvements in Equal Error Rate (EER) over the state-of-the-art methods on the ARL Face Database, Visible and Thermal Paired Face Database, and Tufts Face Database, respectively. An extended dataset (ARL Face Dataset volume III) consisting of polarimetric thermal faces of 121 subjects is also introduced in this paper. Furthermore, an ablation study is conducted to demonstrate the effectiveness of different modules in the proposed method.
- 18 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 3 (2) , 266-280
Multi-Scale Thermal to Visible Face Verification via Attribute Guided Synthesis
Thermal-to-visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or learn domain-invariant robust features from these modalities for cross-modal matching. In this paper, we use attributes extracted from visible images to synthesize attribute-preserved visible images from thermal imagery for cross-modal matching. A pre-trained attribute predictor network is used to extract the attributes from the visible image. Then, a novel multi-scale generator is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. Finally, a pre-trained VGG-Face network is leveraged to extract features from the synthesized image and the input visible image for verification. Extensive experiments evaluated on three datasets (ARL Face Database, Visible and Thermal Paired Face Database, and Tufts Face Database) demonstrate that the proposed method achieves state-of-the-art performance. In particular, it achieve around 2.41%, 2.85% and 1.77% improvements in Equal Error Rate (EER) over the state-of-the-art methods on the ARL Face Database, Visible and Thermal Paired Face Database, and Tufts Face Database, respectively. An extended dataset (ARL Face Dataset volume III) consisting of polarimetric thermal faces of 121 subjects is also introduced in this paper. Furthermore, an ablation study is conducted to demonstrate the effectiveness of different modules in the proposed method.
- 24 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 21 (2) , 1885-1896
Robust Multi-Modal Sensor Fusion: An Adversarial Approach
In recent years, multi-modal fusion has attracted a lot of research interest, both in academia, and in industry. Multimodal fusion entails the combination of information from a set of different types of sensors. Exploiting complementary information from different sensors, we show that target detection and classification problems can greatly benefit from this fusion approach and result in a performance increase. To achieve this gain, the information fusion from various sensors is shown to require some principled strategy to ensure that additional information is constructively used, and has a positive impact on performance. We subsequently demonstrate the viability of the proposed fusion approach by weakening the strong dependence on the functionality of all sensors, hence introducing additional flexibility in our solution and lifting the severe limitation in unconstrained surveillance settings with potential environmental impact. Our proposed data driven approach to multimodal fusion, exploits selected optimal features from an estimated latent space of data across all modalities. This hidden space is learned via a generative network conditioned on individual sensor modalities. The hidden space, as an intrinsic structure, is then exploited in detecting damaged sensors, and in subsequently safeguarding the performance of the fused sensor system. Experimental results show that such an approach can achieve automatic system robustness against noisy/damaged sensors.
- 24 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 21 (2) , 1885-1896
Robust Multi-Modal Sensor Fusion: An Adversarial Approach
In recent years, multi-modal fusion has attracted a lot of research interest, both in academia, and in industry. Multimodal fusion entails the combination of information from a set of different types of sensors. Exploiting complementary information from different sensors, we show that target detection and classification problems can greatly benefit from this fusion approach and result in a performance increase. To achieve this gain, the information fusion from various sensors is shown to require some principled strategy to ensure that additional information is constructively used, and has a positive impact on performance. We subsequently demonstrate the viability of the proposed fusion approach by weakening the strong dependence on the functionality of all sensors, hence introducing additional flexibility in our solution and lifting the severe limitation in unconstrained surveillance settings with potential environmental impact. Our proposed data driven approach to multimodal fusion, exploits selected optimal features from an estimated latent space of data across all modalities. This hidden space is learned via a generative network conditioned on individual sensor modalities. The hidden space, as an intrinsic structure, is then exploited in detecting damaged sensors, and in subsequently safeguarding the performance of the fused sensor system. Experimental results show that such an approach can achieve automatic system robustness against noisy/damaged sensors.
- Open Access
- 1 March 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 3, 1160-1169
An Order Preserving Bilinear Model for Person Detection in Multi-Modal Data
We propose a new order preserving bilinear framework that exploits low-resolution video for person detection in a multi-modal setting using deep neural networks. In this setting cameras are strategically placed such that less robust sensors, e.g. geophones that monitor seismic activity, are located within the field of views (FOVs) of cameras. The primary challenge is being able to leverage sufficient information from videos where there are less than 40 pixels on targets, while also taking advantage of less discriminative information from other modalities, e.g. seismic. Unlike state-of-the-art methods, our bilinear framework retains spatio-temporal order when computing the vector outer products between pairs of features. Despite the high dimensionality of these outer products, we demonstrate that our order preserving bilinear framework yields better performance than recent orderless bilinear models and alternative fusion methods. Code is available at https://github.com/oulutan/OP-Bilinear-Model.
- 1 March 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 3, 1160-1169
Open AccessAn Order Preserving Bilinear Model for Person Detection in Multi-Modal Data
We propose a new order preserving bilinear framework that exploits low-resolution video for person detection in a multi-modal setting using deep neural networks. In this setting cameras are strategically placed such that less robust sensors, e.g. geophones that monitor seismic activity, are located within the field of views (FOVs) of cameras. The primary challenge is being able to leverage sufficient information from videos where there are less than 40 pixels on targets, while also taking advantage of less discriminative information from other modalities, e.g. seismic. Unlike state-of-the-art methods, our bilinear framework retains spatio-temporal order when computing the vector outer products between pairs of features. Despite the high dimensionality of these outer products, we demonstrate that our order preserving bilinear framework yields better performance than recent orderless bilinear models and alternative fusion methods. Code is available at https://github.com/oulutan/OP-Bilinear-Model.