Scott Friedman
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Scott Friedman has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
Aadarsh Swaminathan
Department of Computational Mathematics, Science, & Engineering Michigan State University
2 shared publications- Ashlin Riggs
Department of Computational Mathematics, Science, & Engineering Michigan State University
2 shared publications - Jeff Rye
SIFT Minneapolis MN USA
2 shared publications - Kuldeep Singh
Michigan State University Michigan State University
2 shared publications - Maryam Berijanian
Michigan State University Michigan State University
2 shared publications - Michael Riley Millikan
Department of Computational Mathematics, Science, & Engineering Michigan State University
2 shared publications
Latest Publications
- Open Access
Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis
Modern models for common NLP tasks often employ machine learning techniques and train on journalistic, social media, or other culturally-derived text. These have recently been scrutinized for racial and gender biases, rooting from inherent bias in their training text. These biases are often sub-optimal and recent work poses methods to rectify them; however, these biases may shed light on actual racial or gender gaps in the culture(s) that produced the training text, thereby helping us understand cultural context through big data. This paper presents an approach for quantifying gender bias in word embeddings, and then using them to characterize statistical gender gaps in education, politics, economics, and health. We validate these metrics on 2018 Twitter data spanning 51 U.S. regions and 99 countries. We correlate state and country word embedding biases with 18 international and 5 U.S.-based statistical gender gaps, characterizing regularities and predictive strength.
Open AccessRelating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis
Modern models for common NLP tasks often employ machine learning techniques and train on journalistic, social media, or other culturally-derived text. These have recently been scrutinized for racial and gender biases, rooting from inherent bias in their training text. These biases are often sub-optimal and recent work poses methods to rectify them; however, these biases may shed light on actual racial or gender gaps in the culture(s) that produced the training text, thereby helping us understand cultural context through big data. This paper presents an approach for quantifying gender bias in word embeddings, and then using them to characterize statistical gender gaps in education, politics, economics, and health. We validate these metrics on 2018 Twitter data spanning 51 U.S. regions and 99 countries. We correlate state and country word embedding biases with 18 international and 5 U.S.-based statistical gender gaps, characterizing regularities and predictive strength.
- Open Access
Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs
Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives, influenced heavily by their prompts and training data. As part of LLM adoption, we seek to characterize - and ideally, manage - the socio-cultural values that they express, for reasons of safety, accuracy, inclusion, and cultural fidelity. We present a validated approach to automatically (1) extracting heterogeneous latent value propositions from texts, (2) assessing resonance and conflict of values with texts, and (3) combining these operations to characterize the pluralistic value alignment of human-sourced and LLM-sourced textual data.
Open AccessBottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs
Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives, influenced heavily by their prompts and training data. As part of LLM adoption, we seek to characterize - and ideally, manage - the socio-cultural values that they express, for reasons of safety, accuracy, inclusion, and cultural fidelity. We present a validated approach to automatically (1) extracting heterogeneous latent value propositions from texts, (2) assessing resonance and conflict of values with texts, and (3) combining these operations to characterize the pluralistic value alignment of human-sourced and LLM-sourced textual data.
- Open Access
Soft Measures for Extracting Causal Collective Intelligence
Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.
Open AccessSoft Measures for Extracting Causal Collective Intelligence
Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.
- 9 July 2024
- Published by Springer Nature in Journal of Racial and Ethnic Health Disparities
- Vol. 12 (4) , 2675-2685
Racial Differences in Stigmatizing and Positive Language in Emergency Medicine Notes
Objective Language used by providers in medical documentation may reveal evidence of race-related implicit bias. We aimed to use natural language processing (NLP) to examine if prevalence of stigmatizing language in emergency medicine (EM) encounter notes differs across patient race/ethnicity. Methods In a retrospective cohort of EM encounters, NLP techniques identified stigmatizing and positive themes. Logistic regression models analyzed the association of race/ethnicity and themes within notes. Outcomes were the presence (or absence) of 7 different themes: 5 stigmatizing (difficult, non-compliant, skepticism, substance abuse/seeking, and financial difficulty) and 2 positive (compliment and compliant). Results The sample included notes from 26,363 unique patients. NH Black patient notes were less likely to contain difficult (odds ratio (OR) 0.80, 95% confidence interval (CI), 0.73–0.88), skepticism (OR 0.87, 95% CI, 0.79–0.96), and substance abuse/seeking (OR 0.62, 95% CI, 0.56–0.70) compared to NH White patient notes but more likely to contain non-compliant (OR 1.26, 95% CI, 1.17–1.36) and financial difficulty (OR 1.14, 95% CI, 1.04–1.25). Hispanic patient notes were less likely to contain difficult (OR 0.68, 95% CI, 0.58–0.80) and substance abuse/seeking (OR 0.78, 95% CI, 0.66–0.93). NH NA/AI patient notes had twice the odds as NH White patient notes to contain a stigmatizing theme (OR 2.02, 95% CI, 1.64–2.49). Conclusions Using an NLP model to analyze themes in EM notes across racial groups, we identified several inequities in the usage of positive and stigmatizing language. Interventions to minimize race-related implicit bias should be undertaken.
- 9 July 2024
- Published by Springer Nature in Journal of Racial and Ethnic Health Disparities
- Vol. 12 (4) , 2675-2685
Racial Differences in Stigmatizing and Positive Language in Emergency Medicine Notes
Objective Language used by providers in medical documentation may reveal evidence of race-related implicit bias. We aimed to use natural language processing (NLP) to examine if prevalence of stigmatizing language in emergency medicine (EM) encounter notes differs across patient race/ethnicity. Methods In a retrospective cohort of EM encounters, NLP techniques identified stigmatizing and positive themes. Logistic regression models analyzed the association of race/ethnicity and themes within notes. Outcomes were the presence (or absence) of 7 different themes: 5 stigmatizing (difficult, non-compliant, skepticism, substance abuse/seeking, and financial difficulty) and 2 positive (compliment and compliant). Results The sample included notes from 26,363 unique patients. NH Black patient notes were less likely to contain difficult (odds ratio (OR) 0.80, 95% confidence interval (CI), 0.73–0.88), skepticism (OR 0.87, 95% CI, 0.79–0.96), and substance abuse/seeking (OR 0.62, 95% CI, 0.56–0.70) compared to NH White patient notes but more likely to contain non-compliant (OR 1.26, 95% CI, 1.17–1.36) and financial difficulty (OR 1.14, 95% CI, 1.04–1.25). Hispanic patient notes were less likely to contain difficult (OR 0.68, 95% CI, 0.58–0.80) and substance abuse/seeking (OR 0.78, 95% CI, 0.66–0.93). NH NA/AI patient notes had twice the odds as NH White patient notes to contain a stigmatizing theme (OR 2.02, 95% CI, 1.64–2.49). Conclusions Using an NLP model to analyze themes in EM notes across racial groups, we identified several inequities in the usage of positive and stigmatizing language. Interventions to minimize race-related implicit bias should be undertaken.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 99-116
Soft Measures for Extracting Causal Collective Intelligence
Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone. Proceedings of the 1st Workshop on NLP for Science (NLP4Science). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 99-116
Open AccessSoft Measures for Extracting Causal Collective Intelligence
Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone. Proceedings of the 1st Workshop on NLP for Science (NLP4Science). 2024.
- Open Access
Provenance-Based Interpretation of Multi-Agent Information Analysis
Analytic software tools and workflows are increasing in capability, complexity, number, and scale, and the integrity of our workflows is as important as ever. Specifically, we must be able to inspect the process of analytic workflows to assess (1) confidence of the conclusions, (2) risks and biases of the operations involved, (3) sensitivity of the conclusions to sources and agents, (4) impact and pertinence of various sources and agents, and (5) diversity of the sources that support the conclusions. We present an approach that tracks agents' provenance with PROV-O in conjunction with agents' appraisals and evidence links (expressed in our novel DIVE ontology). Together, PROV-O and DIVE enable dynamic propagation of confidence and counter-factual refutation to improve human-machine trust and analytic integrity. We demonstrate representative software developed for user interaction with that provenance, and discuss key needs for organizations adopting such approaches. We demonstrate all of these assessments in a multi-agent analysis scenario, using an interactive web-based information validation UI.
Open AccessProvenance-Based Interpretation of Multi-Agent Information Analysis
Analytic software tools and workflows are increasing in capability, complexity, number, and scale, and the integrity of our workflows is as important as ever. Specifically, we must be able to inspect the process of analytic workflows to assess (1) confidence of the conclusions, (2) risks and biases of the operations involved, (3) sensitivity of the conclusions to sources and agents, (4) impact and pertinence of various sources and agents, and (5) diversity of the sources that support the conclusions. We present an approach that tracks agents' provenance with PROV-O in conjunction with agents' appraisals and evidence links (expressed in our novel DIVE ontology). Together, PROV-O and DIVE enable dynamic propagation of confidence and counter-factual refutation to improve human-machine trust and analytic integrity. We demonstrate representative software developed for user interaction with that provenance, and discuss key needs for organizations adopting such approaches. We demonstrate all of these assessments in a multi-agent analysis scenario, using an interactive web-based information validation UI.
- Open Access
Combining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise
Many modern machine learning approaches require vast amounts of training data to learn new concepts; conversely, human learning often requires few examples--sometimes only one--from which the learner can abstract structural concepts. We present a novel approach to introducing new spatial structures to an AI agent, combining deep learning over qualitative spatial relations with various heuristic search algorithms. The agent extracts spatial relations from a sparse set of noisy examples of block-based structures, and trains convolutional and sequential models of those relation sets. To create novel examples of similar structures, the agent begins placing blocks on a virtual table, uses a CNN to predict the most similar complete example structure after each placement, an LSTM to predict the most likely set of remaining moves needed to complete it, and recommends one using heuristic search. We verify that the agent learned the concept by observing its virtual block-building activities, wherein it ranks each potential subsequent action toward building its learned concept. We empirically assess this approach with human participants' ratings of the block structures. Initial results and qualitative evaluations of structures generated by the trained agent show where it has generalized concepts from the training data, which heuristics perform best within the search space, and how we might improve learning and execution.
Open AccessCombining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise
Many modern machine learning approaches require vast amounts of training data to learn new concepts; conversely, human learning often requires few examples--sometimes only one--from which the learner can abstract structural concepts. We present a novel approach to introducing new spatial structures to an AI agent, combining deep learning over qualitative spatial relations with various heuristic search algorithms. The agent extracts spatial relations from a sparse set of noisy examples of block-based structures, and trains convolutional and sequential models of those relation sets. To create novel examples of similar structures, the agent begins placing blocks on a virtual table, uses a CNN to predict the most similar complete example structure after each placement, an LSTM to predict the most likely set of remaining moves needed to complete it, and recommends one using heuristic search. We verify that the agent learned the concept by observing its virtual block-building activities, wherein it ranks each potential subsequent action toward building its learned concept. We empirically assess this approach with human participants' ratings of the block structures. Initial results and qualitative evaluations of structures generated by the trained agent show where it has generalized concepts from the training data, which heuristics perform best within the search space, and how we might improve learning and execution.
- Open Access
- 27 December 2017
- Published by Wiley in Cognitive Science
- Vol. 42 (4) , 1110-1145
Representing, Running, and Revising Mental Models: A Computational Model
People use commonsense science knowledge to flexibly explain, predict, and manipulate the world around them, yet we lack computational models of how this commonsense science knowledge is represented, acquired, utilized, and revised. This is an important challenge for cognitive science: Building higher order computational models in this area will help characterize one of the hallmarks of human reasoning, and it will allow us to build more robust reasoning systems. This paper presents a novel assembled coherence (AC) theory of human conceptual change, whereby people revise beliefs and mental models by constructing and evaluating explanations using fragmentary, globally inconsistent knowledge. We implement AC theory with Timber, a computational model of conceptual change that revises its beliefs and generates human‐like explanations in commonsense science. Timber represents domain knowledge using predicate calculus and qualitative model fragments, and uses an abductive model formulation algorithm to construct competing explanations for phenomena. Timber then (a) scores competing explanations with respect to previously accepted beliefs, using a cost function based on simplicity and credibility, (b) identifies a low‐cost, preferred explanation and accepts its constituent beliefs, and then (c) greedily alters previous explanation preferences to reduce global cost and thereby revise beliefs. Consistency is a soft constraint in Timber; it is biased to select explanations that share consistent beliefs, assumptions, and causal structure with its other, preferred explanations. In this paper, we use Timber to simulate the belief changes of students during clinical interviews about how the seasons change. We show that Timber produces and revises a sequence of explanations similar to those of the students, which supports the psychological plausibility of AC theory.
- 27 December 2017
- Published by Wiley in Cognitive Science
- Vol. 42 (4) , 1110-1145
Open AccessRepresenting, Running, and Revising Mental Models: A Computational Model
People use commonsense science knowledge to flexibly explain, predict, and manipulate the world around them, yet we lack computational models of how this commonsense science knowledge is represented, acquired, utilized, and revised. This is an important challenge for cognitive science: Building higher order computational models in this area will help characterize one of the hallmarks of human reasoning, and it will allow us to build more robust reasoning systems. This paper presents a novel assembled coherence (AC) theory of human conceptual change, whereby people revise beliefs and mental models by constructing and evaluating explanations using fragmentary, globally inconsistent knowledge. We implement AC theory with Timber, a computational model of conceptual change that revises its beliefs and generates human‐like explanations in commonsense science. Timber represents domain knowledge using predicate calculus and qualitative model fragments, and uses an abductive model formulation algorithm to construct competing explanations for phenomena. Timber then (a) scores competing explanations with respect to previously accepted beliefs, using a cost function based on simplicity and credibility, (b) identifies a low‐cost, preferred explanation and accepts its constituent beliefs, and then (c) greedily alters previous explanation preferences to reduce global cost and thereby revise beliefs. Consistency is a soft constraint in Timber; it is biased to select explanations that share consistent beliefs, assumptions, and causal structure with its other, preferred explanations. In this paper, we use Timber to simulate the belief changes of students during clinical interviews about how the seasons change. We show that Timber produces and revises a sequence of explanations similar to those of the students, which supports the psychological plausibility of AC theory.