Craig Greenberg
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Craig Greenberg has a total of 38 co-authors. Here are the top co-authors by number of shared publications:
Douglas Reynolds
2 shared publicationsElliot Singer
2 shared publications- Lisa Reyes Mason
Graduate School of Social Work, University of Denver
2 shared publications Seyed Omid Sadjadi
2 shared publicationsAdrian Munoz-Barrera
Genomics Division, Instituto Tecnologico y de Energias Renovables (ITER), Granadilla de Abona, Spain
1 shared publication- Afzal Godil1 shared publication
Latest Publications
- 13 November 2025
- Published by National Institute of Standards and Technology (NIST)
Assessing risks and impacts of AI (ARIA) :
This document describes the procedure used for a pilot of NIST’s Assessing Risks and Impacts of AI (ARIA) evaluation: ARIA 0.1. Five organizations participated, submitting a total of 7 AI applications to be evaluated. In this document, we first describe the design of the three evaluation scenarios (TV Spoilers, Meal Planner, Pathfinder) and the three testing levels (model testing, red teaming, field testing). We then discuss the methods used for assessment via dialogue annotation and tester questionnaires. Finally, we describe our approach to measuring validity of AI applications using measurement trees.
- 13 November 2025
- Published by National Institute of Standards and Technology (NIST)
Assessing risks and impacts of AI (ARIA) :
This document describes the procedure used for a pilot of NIST’s Assessing Risks and Impacts of AI (ARIA) evaluation: ARIA 0.1. Five organizations participated, submitting a total of 7 AI applications to be evaluated. In this document, we first describe the design of the three evaluation scenarios (TV Spoilers, Meal Planner, Pathfinder) and the three testing levels (model testing, red teaming, field testing). We then discuss the methods used for assessment via dialogue annotation and tester questionnaires. Finally, we describe our approach to measuring validity of AI applications using measurement trees.
- Open Access
Small variant benchmark from a complete assembly of X and Y chromosomes
The sex chromosomes contain complex, important genes impacting medical phenotypes, but differ from the autosomes in their ploidy and large repetitive regions. To evaluate variant detection on chromosomes X and Y, we created an 111,725 variant benchmark for the Genome in a Bottle HG002 reference material. We show how complete assemblies can expand benchmarks to difficult regions, but highlight remaining challenges benchmarking complex gene conversions, copy number variable gene arrays, and human satellites.
Open AccessSmall variant benchmark from a complete assembly of X and Y chromosomes
The sex chromosomes contain complex, important genes impacting medical phenotypes, but differ from the autosomes in their ploidy and large repetitive regions. To evaluate variant detection on chromosomes X and Y, we created an 111,725 variant benchmark for the Genome in a Bottle HG002 reference material. We show how complete assemblies can expand benchmarks to difficult regions, but highlight remaining challenges benchmarking complex gene conversions, copy number variable gene arrays, and human satellites.
- Open Access
The 2021 NIST Speaker Recognition Evaluation
The 2021 Speaker Recognition Evaluation (SRE21) was the latest cycle of the ongoing evaluation series conducted by the U.S. National Institute of Standards and Technology (NIST) since 1996. It was the second large-scale multimodal speaker/person recognition evaluation organized by NIST (the first one being SRE19). Similar to SRE19, it featured two core evaluation tracks, namely audio and audio-visual, as well as an optional visual track. In addition to offering fixed and open training conditions, it also introduced new challenges for the community, thanks to a new multimodal (i.e., audio, video, and selfie images) and multilingual (i.e., with multilingual speakers) corpus, termed WeCanTalk, collected outside North America by the Linguistic Data Consortium (LDC). These challenges included: 1) trials (target and non-target) with enrollment and test segments originating from different domains (i.e., telephony versus video), and 2) trials (target and non-target) with enrollment and test segments spoken in different languages (i.e., cross-lingual trials). This paper presents an overview of SRE21 including the tasks, performance metric, data, evaluation protocol, results and system performance analyses. A total of 23 organizations (forming 15 teams) from academia and industry participated in SRE21 and submitted 158 valid system outputs. Evaluation results indicate: audio-visual fusion produce substantial gains in performance over audio-only or visual-only systems; top performing speaker and face recognition systems exhibited comparable performance under the matched domain conditions present in this evaluation; and, the use of complex neural network architectures (e.g., ResNet) along with angular losses with margin, data augmentation, as well as long duration fine-tuning contributed to notable performance improvements for the audio-only speaker recognition task.
Open AccessThe 2021 NIST Speaker Recognition Evaluation
The 2021 Speaker Recognition Evaluation (SRE21) was the latest cycle of the ongoing evaluation series conducted by the U.S. National Institute of Standards and Technology (NIST) since 1996. It was the second large-scale multimodal speaker/person recognition evaluation organized by NIST (the first one being SRE19). Similar to SRE19, it featured two core evaluation tracks, namely audio and audio-visual, as well as an optional visual track. In addition to offering fixed and open training conditions, it also introduced new challenges for the community, thanks to a new multimodal (i.e., audio, video, and selfie images) and multilingual (i.e., with multilingual speakers) corpus, termed WeCanTalk, collected outside North America by the Linguistic Data Consortium (LDC). These challenges included: 1) trials (target and non-target) with enrollment and test segments originating from different domains (i.e., telephony versus video), and 2) trials (target and non-target) with enrollment and test segments spoken in different languages (i.e., cross-lingual trials). This paper presents an overview of SRE21 including the tasks, performance metric, data, evaluation protocol, results and system performance analyses. A total of 23 organizations (forming 15 teams) from academia and industry participated in SRE21 and submitted 158 valid system outputs. Evaluation results indicate: audio-visual fusion produce substantial gains in performance over audio-only or visual-only systems; top performing speaker and face recognition systems exhibited comparable performance under the matched domain conditions present in this evaluation; and, the use of complex neural network architectures (e.g., ResNet) along with angular losses with margin, data augmentation, as well as long duration fine-tuning contributed to notable performance improvements for the audio-only speaker recognition task.
- Open Access
The NIST CTS Speaker Recognition Challenge
The US National Institute of Standards and Technology (NIST) has been conducting a second iteration of the CTS challenge since August 2020. The current iteration of the CTS Challenge is a leaderboard-style speaker recognition evaluation using telephony data extracted from the unexposed portions of the Call My Net 2 (CMN2) and Multi-Language Speech (MLS) corpora collected by the LDC. The CTS Challenge is currently organized in a similar manner to the SRE19 CTS Challenge, offering only an open training condition using two evaluation subsets, namely Progress and Test. Unlike in the SRE19 Challenge, no training or development set was initially released, and NIST has publicly released the leaderboards on both subsets for the CTS Challenge. Which subset (i.e., Progress or Test) a trial belongs to is unknown to challenge participants, and each system submission needs to contain outputs for all of the trials. The CTS Challenge has also served, and will continue to do so, as a prerequisite for entrance to the regular SREs (such as SRE21). Since August 2020, a total of 53 organizations (forming 33 teams) from academia and industry have participated in the CTS Challenge and submitted more than 4400 valid system outputs. This paper presents an overview of the evaluation and several analyses of system performance for some primary conditions in the CTS Challenge. The CTS Challenge results thus far indicate remarkable improvements in performance due to 1) speaker embeddings extracted using large-scale and complex neural network architectures such as ResNets along with angular margin losses for speaker embedding extraction, 2) extensive data augmentation, 3) the use of large amounts of in-house proprietary data from a large number of labeled speakers, 4) long-duration fine-tuning.
Open AccessThe NIST CTS Speaker Recognition Challenge
The US National Institute of Standards and Technology (NIST) has been conducting a second iteration of the CTS challenge since August 2020. The current iteration of the CTS Challenge is a leaderboard-style speaker recognition evaluation using telephony data extracted from the unexposed portions of the Call My Net 2 (CMN2) and Multi-Language Speech (MLS) corpora collected by the LDC. The CTS Challenge is currently organized in a similar manner to the SRE19 CTS Challenge, offering only an open training condition using two evaluation subsets, namely Progress and Test. Unlike in the SRE19 Challenge, no training or development set was initially released, and NIST has publicly released the leaderboards on both subsets for the CTS Challenge. Which subset (i.e., Progress or Test) a trial belongs to is unknown to challenge participants, and each system submission needs to contain outputs for all of the trials. The CTS Challenge has also served, and will continue to do so, as a prerequisite for entrance to the regular SREs (such as SRE21). Since August 2020, a total of 53 organizations (forming 33 teams) from academia and industry have participated in the CTS Challenge and submitted more than 4400 valid system outputs. This paper presents an overview of the evaluation and several analyses of system performance for some primary conditions in the CTS Challenge. The CTS Challenge results thus far indicate remarkable improvements in performance due to 1) speaker embeddings extracted using large-scale and complex neural network architectures such as ResNets along with angular margin losses for speaker embedding extraction, 2) extensive data augmentation, 3) the use of large amounts of in-house proprietary data from a large number of labeled speakers, 4) long-duration fine-tuning.