John J. Howard
Research Subjects
Research Interests
John J. Howard has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Yevgeniy B. Sirotin
The Identity and Data Sciences Lab at The Maryland Test Facility
8 shared publications - Cynthia M. Cook
The Identity and Data Sciences Laboratory (IDSL), SAIC
3 shared publications Arun R. Vemury
United States Department of Homeland Security, Science and Technology Directorate, Washington, DC, USA
2 shared publications- Arun R. Vemury
Science and Technology Directorate, US Department of Homeland Security
2 shared publications - Eli Laird
Southern Methodist University
2 shared publications Jerry L. Tipton
Management Department, Maryland Test Facility, Upper Malboro, MA, USA
2 shared publications
Latest Publications
- Open Access
- 30 June 2025
- Published by Association for Computing Machinery (ACM) in ACM Journal on Responsible Computing
- Vol. 2 (2) , 1-23
Colorimetric skin tone scale for improved accuracy of human skin tone annotations
Human image datasets used to develop and evaluate technology should represent the variation in human phenotypes, including skin tone. Datasets that include skin tone information frequently rely on manual skin tone ratings based on the Fitzpatrick Skin Type (FST) or the Monk Skin Tone (MST) scales in lieu of the actual measured skin tone of the image dataset subjects. However, perceived skin tone is subject to known perceptual errors and skin tone appearance in digital images can vary substantially depending on the capture camera and environment, confounding manual ratings. Surprisingly, the relationship between skin tone ratings and measured skin tone has not been explored. To close this research gap, we measured the relationship between skin tone ratings from existing scales (FST, MST) and skin tone values measured by a calibrated colorimeter. We also propose and assess a novel Colorimetric Skin Tone (CST) scale developed based on prior colorimetric measurements. Using experiments requiring humans to rate their own skin tone and the skin tone of subjects in images, we show that the new CST scale is more sensitive, consistent, and colorimetrically accurate. While skin tone ratings appeared to correct for some color variation across images, they introduced perceptual errors related to race and other factors. These perceptual errors must be considered before using manual skin-tone ratings in technology evaluations or for engineering decisions.
- 30 June 2025
- Published by Association for Computing Machinery (ACM) in ACM Journal on Responsible Computing
- Vol. 2 (2) , 1-23
Open AccessColorimetric skin tone scale for improved accuracy of human skin tone annotations
Human image datasets used to develop and evaluate technology should represent the variation in human phenotypes, including skin tone. Datasets that include skin tone information frequently rely on manual skin tone ratings based on the Fitzpatrick Skin Type (FST) or the Monk Skin Tone (MST) scales in lieu of the actual measured skin tone of the image dataset subjects. However, perceived skin tone is subject to known perceptual errors and skin tone appearance in digital images can vary substantially depending on the capture camera and environment, confounding manual ratings. Surprisingly, the relationship between skin tone ratings and measured skin tone has not been explored. To close this research gap, we measured the relationship between skin tone ratings from existing scales (FST, MST) and skin tone values measured by a calibrated colorimeter. We also propose and assess a novel Colorimetric Skin Tone (CST) scale developed based on prior colorimetric measurements. Using experiments requiring humans to rate their own skin tone and the skin tone of subjects in images, we show that the new CST scale is more sensitive, consistent, and colorimetrically accurate. While skin tone ratings appeared to correct for some color variation across images, they introduced perceptual errors related to race and other factors. These perceptual errors must be considered before using manual skin-tone ratings in technology evaluations or for engineering decisions.
- Open Access
- 23 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 2630-2639
Performance Differentials in Deployed Biometric Systems Caused by Open-Source Face Detectors
- 23 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 2630-2639
Open AccessPerformance Differentials in Deployed Biometric Systems Caused by Open-Source Face Detectors
- 30 July 2023
- Published by Springer Nature
- p. 431-447
Evaluating Proposed Fairness Models for Face Recognition Algorithms
- 30 July 2023
- Published by Springer Nature
- p. 431-447
Evaluating Proposed Fairness Models for Face Recognition Algorithms
- 30 July 2023
- Published by Springer Nature
- p. 448-464
Disparate Impact in Facial Recognition Stems from the Broad Homogeneity Effect: A Case Study and Method to Resolve
- 30 July 2023
- Published by Springer Nature
- p. 448-464
Disparate Impact in Facial Recognition Stems from the Broad Homogeneity Effect: A Case Study and Method to Resolve
- 27 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 3 (4) , 550-560
Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics
With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups, such as race, age, and gender. Recently, phenotypes such as skin tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin tone in evaluations of biometric performance or in AI more broadly. Biometric researchers have estimated skin tone, most notably focusing on face area lightness measures (FALMs) using automated color analysis or Fitzpatrick Skin Types (FST). These estimates have generally been based on the same images used to assess biometric performance, which are often collected using unknown and varied devices, at unknown and varied times, and under unknown and varied environmental conditions. In this study, we explore the relationship between FALMs estimated from images and ground-truth skin readings collected using a colormeter device specifically designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALMs. This variation was only reduced by greater control of acquisition (camera, background, and environmental conditions). Next, we compare ground-truth FALMs to FST categories obtained using the standard, in-person, medical survey. We found that there was relatively little change in ground-truth FALMs across different FST category and that FST correlated more with self-reported race than with ground-truth FALMs. These findings show FST is poorly predictive of skin tone and should not be used as such in evaluations of computer vision applications. Finally, using modeling, we show that when face recognition performance is driven by FALMs and independent of race, noisy FALM estimates can lead to erroneous selection of race as a key correlate of biometric performance. These results demonstrate that measures of skin type for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin tone.
- 27 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biometrics, Behavior, and Identity Science
- Vol. 3 (4) , 550-560
Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics
With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups, such as race, age, and gender. Recently, phenotypes such as skin tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin tone in evaluations of biometric performance or in AI more broadly. Biometric researchers have estimated skin tone, most notably focusing on face area lightness measures (FALMs) using automated color analysis or Fitzpatrick Skin Types (FST). These estimates have generally been based on the same images used to assess biometric performance, which are often collected using unknown and varied devices, at unknown and varied times, and under unknown and varied environmental conditions. In this study, we explore the relationship between FALMs estimated from images and ground-truth skin readings collected using a colormeter device specifically designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALMs. This variation was only reduced by greater control of acquisition (camera, background, and environmental conditions). Next, we compare ground-truth FALMs to FST categories obtained using the standard, in-person, medical survey. We found that there was relatively little change in ground-truth FALMs across different FST category and that FST correlated more with self-reported race than with ground-truth FALMs. These findings show FST is poorly predictive of skin tone and should not be used as such in evaluations of computer vision applications. Finally, using modeling, we show that when face recognition performance is driven by FALMs and independent of race, noisy FALM estimates can lead to erroneous selection of race as a key correlate of biometric performance. These results demonstrate that measures of skin type for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin tone.
- Open Access
- 21 August 2020
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 15 (8) , e0237855
Human-algorithm teaming in face recognition: How algorithm outcomes cognitively bias human decision-making
In face recognition applications, humans often team with algorithms, reviewing algorithm results to make an identity decision. However, few studies have explicitly measured how algorithms influence human face matching performance. One study that did examine this interaction found a concerning deterioration of human accuracy in the presence of algorithm errors. We conducted an experiment to examine how prior face identity decisions influence subsequent human judgements about face similarity. 376 volunteers were asked to rate the similarity of face pairs along a scale. Volunteers performing the task were told that they were reviewing identity decisions made by different sources, either a computer or human, or were told to make their own judgement without prior information. Replicating past results, we found that prior identity decisions, presented as labels, influenced volunteers’ own identity judgements. We extend these results as follows. First, we show that the influence of identity decision labels was independent of indicated decision source (human or computer) despite volunteers’ greater distrust of human identification ability. Second, applying a signal detection theory framework, we show that prior identity decision labels did not reduce volunteers’ attention to the face pair. Discrimination performance was the same with and without the labels. Instead, prior identity decision labels altered volunteers’ internal criterion used to judge a face pair as “matching” or “non-matching”. This shifted volunteers’ face pair similarity judgements by a full step along the response scale. Our work shows how human face matching is affected by prior identity decision labels and we discuss how this may limit the total accuracy of human-algorithm teams performing face matching tasks.
- 21 August 2020
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 15 (8) , e0237855
Open AccessHuman-algorithm teaming in face recognition: How algorithm outcomes cognitively bias human decision-making
In face recognition applications, humans often team with algorithms, reviewing algorithm results to make an identity decision. However, few studies have explicitly measured how algorithms influence human face matching performance. One study that did examine this interaction found a concerning deterioration of human accuracy in the presence of algorithm errors. We conducted an experiment to examine how prior face identity decisions influence subsequent human judgements about face similarity. 376 volunteers were asked to rate the similarity of face pairs along a scale. Volunteers performing the task were told that they were reviewing identity decisions made by different sources, either a computer or human, or were told to make their own judgement without prior information. Replicating past results, we found that prior identity decisions, presented as labels, influenced volunteers’ own identity judgements. We extend these results as follows. First, we show that the influence of identity decision labels was independent of indicated decision source (human or computer) despite volunteers’ greater distrust of human identification ability. Second, applying a signal detection theory framework, we show that prior identity decision labels did not reduce volunteers’ attention to the face pair. Discrimination performance was the same with and without the labels. Instead, prior identity decision labels altered volunteers’ internal criterion used to judge a face pair as “matching” or “non-matching”. This shifted volunteers’ face pair similarity judgements by a full step along the response scale. Our work shows how human face matching is affected by prior identity decision labels and we discuss how this may limit the total accuracy of human-algorithm teams performing face matching tasks.
- 1 October 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-4
Operational Tradeoffs in the 2018 Department of Homeland Security Science and Technology Directorate Biometric Technology Rally
The 2018 Biometric Technology Rally was an evaluation, sponsored by the U.S. Department of Homeland Security (DHS), Science and Technology (S&T) Directorate, that challenged industry to provide face or face/iris systems capable of unmanned, traveler identification in a high-throughput security environment. Eleven selected systems were installed at the Maryland Test Facility (MdTF), a DHS S&T affiliated biometrics testing laboratory, and evaluated using a sample of 363 naive human subjects recruited from the general public. The performance of each system was examined based on measured throughput (efficiency), matching capability (effectiveness), and user satisfaction. This research documents the operational tradeoffs between these three measures of system performance. Specifically, we perform two tradeoff analyses: efficiency versus effectiveness and satisfaction versus both efficiency and effectiveness. These tradeoff analyses allow us to determine how and if these three performance measures are related in the various kinds of biometric systems we tested. For example, are higher throughput systems also more effective? Do people prefer systems that are faster or more effective? Our results show there is no clear relationship between how quickly a system can process a user and how well it can identify the user. Furthermore, there was also no significant relationship observed between how quickly a system can process a user and how satisfied the user is with the system. However, there was a strong relationship between how well a system identifies a user and how satisfied the user is with the system. These outcomes suggest that some systems could benefit by leveraging additional collection time to collect a higher quality image. Users did not tend to prefer faster systems but did prefer a system they thought was working as intended. Finally, these results also show that in regards to public acceptance, systems designers should focus on correctly identifying larger populations of users rather than how quickly a given user can be processed.
- 1 October 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-4
Operational Tradeoffs in the 2018 Department of Homeland Security Science and Technology Directorate Biometric Technology Rally
The 2018 Biometric Technology Rally was an evaluation, sponsored by the U.S. Department of Homeland Security (DHS), Science and Technology (S&T) Directorate, that challenged industry to provide face or face/iris systems capable of unmanned, traveler identification in a high-throughput security environment. Eleven selected systems were installed at the Maryland Test Facility (MdTF), a DHS S&T affiliated biometrics testing laboratory, and evaluated using a sample of 363 naive human subjects recruited from the general public. The performance of each system was examined based on measured throughput (efficiency), matching capability (effectiveness), and user satisfaction. This research documents the operational tradeoffs between these three measures of system performance. Specifically, we perform two tradeoff analyses: efficiency versus effectiveness and satisfaction versus both efficiency and effectiveness. These tradeoff analyses allow us to determine how and if these three performance measures are related in the various kinds of biometric systems we tested. For example, are higher throughput systems also more effective? Do people prefer systems that are faster or more effective? Our results show there is no clear relationship between how quickly a system can process a user and how well it can identify the user. Furthermore, there was also no significant relationship observed between how quickly a system can process a user and how satisfied the user is with the system. However, there was a strong relationship between how well a system identifies a user and how satisfied the user is with the system. These outcomes suggest that some systems could benefit by leveraging additional collection time to collect a higher quality image. Users did not tend to prefer faster systems but did prefer a system they thought was working as intended. Finally, these results also show that in regards to public acceptance, systems designers should focus on correctly identifying larger populations of users rather than how quickly a given user can be processed.