JHJohn J. Howard

John J. Howard

*This is an algorithmically generated author record based on publicly available information.

John J. Howard has a total of 16 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Colorimetric skin tone scale for improved accuracy of human skin tone annotations

    by Cynthia Cook, John Howard, Laura Rabbitt, Isabelle Shuggi, Yevgeniy Sirotin, Jerry Tipton, Arun Vemury

    Human image datasets used to develop and evaluate technology should represent the variation in human phenotypes, including skin tone. Datasets that include skin tone information frequently rely on manual skin tone ratings based on the Fitzpatrick Skin Type (FST) or the Monk Skin Tone (MST) scales in lieu of the actual measured skin tone of the image dataset subjects. However, perceived skin tone is subject to known perceptual errors and skin tone appearance in digital images can vary substantially depending on the capture camera and environment, confounding manual ratings. Surprisingly, the relationship between skin tone ratings and measured skin tone has not been explored. To close this research gap, we measured the relationship between skin tone ratings from existing scales (FST, MST) and skin tone values measured by a calibrated colorimeter. We also propose and assess a novel Colorimetric Skin Tone (CST) scale developed based on prior colorimetric measurements. Using experiments requiring humans to rate their own skin tone and the skin tone of subjects in images, we show that the new CST scale is more sensitive, consistent, and colorimetrically accurate. While skin tone ratings appeared to correct for some color variation across images, they introduced perceptual errors related to race and other factors. These perceptual errors must be considered before using manual skin-tone ratings in technology evaluations or for engineering decisions.


  • Open Access

    Performance Differentials in Deployed Biometric Systems Caused by Open-Source Face Detectors

    by Cynthia M. Cook, Laurie Cuffney, John J. Howard, Yevgeniy B. Sirotin, Jerry L. Tipton, Arun R. Vemury


  • Evaluating Proposed Fairness Models for Face Recognition Algorithms

    by John J. Howard, Eli J. Laird, Rebecca E. Rubin, Yevgeniy B. Sirotin, Jerry L. Tipton, Arun R. Vemury


  • Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics

    by John J. Howard, Yevgeniy B. Sirotin, Jerry L. Tipton, Arun R. Vemury

    With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups, such as race, age, and gender. Recently, phenotypes such as skin tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin tone in evaluations of biometric performance or in AI more broadly. Biometric researchers have estimated skin tone, most notably focusing on face area lightness measures (FALMs) using automated color analysis or Fitzpatrick Skin Types (FST). These estimates have generally been based on the same images used to assess biometric performance, which are often collected using unknown and varied devices, at unknown and varied times, and under unknown and varied environmental conditions. In this study, we explore the relationship between FALMs estimated from images and ground-truth skin readings collected using a colormeter device specifically designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALMs. This variation was only reduced by greater control of acquisition (camera, background, and environmental conditions). Next, we compare ground-truth FALMs to FST categories obtained using the standard, in-person, medical survey. We found that there was relatively little change in ground-truth FALMs across different FST category and that FST correlated more with self-reported race than with ground-truth FALMs. These findings show FST is poorly predictive of skin tone and should not be used as such in evaluations of computer vision applications. Finally, using modeling, we show that when face recognition performance is driven by FALMs and independent of race, noisy FALM estimates can lead to erroneous selection of race as a key correlate of biometric performance. These results demonstrate that measures of skin type for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin tone.


  • Open Access

    Human-algorithm teaming in face recognition: How algorithm outcomes cognitively bias human decision-making

    by John J. Howard, Laura R. Rabbitt, Yevgeniy B. Sirotin

    In face recognition applications, humans often team with algorithms, reviewing algorithm results to make an identity decision. However, few studies have explicitly measured how algorithms influence human face matching performance. One study that did examine this interaction found a concerning deterioration of human accuracy in the presence of algorithm errors. We conducted an experiment to examine how prior face identity decisions influence subsequent human judgements about face similarity. 376 volunteers were asked to rate the similarity of face pairs along a scale. Volunteers performing the task were told that they were reviewing identity decisions made by different sources, either a computer or human, or were told to make their own judgement without prior information. Replicating past results, we found that prior identity decisions, presented as labels, influenced volunteers’ own identity judgements. We extend these results as follows. First, we show that the influence of identity decision labels was independent of indicated decision source (human or computer) despite volunteers’ greater distrust of human identification ability. Second, applying a signal detection theory framework, we show that prior identity decision labels did not reduce volunteers’ attention to the face pair. Discrimination performance was the same with and without the labels. Instead, prior identity decision labels altered volunteers’ internal criterion used to judge a face pair as “matching” or “non-matching”. This shifted volunteers’ face pair similarity judgements by a full step along the response scale. Our work shows how human face matching is affected by prior identity decision labels and we discuss how this may limit the total accuracy of human-algorithm teams performing face matching tasks.


  • Operational Tradeoffs in the 2018 Department of Homeland Security Science and Technology Directorate Biometric Technology Rally

    by Jacob A. Hasselgren, John J. Howard, Yevgeniy B. Sirotin, Andrew J. Blanchard, Arun Vemury

    The 2018 Biometric Technology Rally was an evaluation, sponsored by the U.S. Department of Homeland Security (DHS), Science and Technology (S&T) Directorate, that challenged industry to provide face or face/iris systems capable of unmanned, traveler identification in a high-throughput security environment. Eleven selected systems were installed at the Maryland Test Facility (MdTF), a DHS S&T affiliated biometrics testing laboratory, and evaluated using a sample of 363 naive human subjects recruited from the general public. The performance of each system was examined based on measured throughput (efficiency), matching capability (effectiveness), and user satisfaction. This research documents the operational tradeoffs between these three measures of system performance. Specifically, we perform two tradeoff analyses: efficiency versus effectiveness and satisfaction versus both efficiency and effectiveness. These tradeoff analyses allow us to determine how and if these three performance measures are related in the various kinds of biometric systems we tested. For example, are higher throughput systems also more effective? Do people prefer systems that are faster or more effective? Our results show there is no clear relationship between how quickly a system can process a user and how well it can identify the user. Furthermore, there was also no significant relationship observed between how quickly a system can process a user and how satisfied the user is with the system. However, there was a strong relationship between how well a system identifies a user and how satisfied the user is with the system. These outcomes suggest that some systems could benefit by leveraging additional collection time to collect a higher quality image. Users did not tend to prefer faster systems but did prefer a system they thought was working as intended. Finally, these results also show that in regards to public acceptance, systems designers should focus on correctly identifying larger populations of users rather than how quickly a given user can be processed.