R. Thomas McCoy
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
R. Thomas McCoy has a total of 26 co-authors. Here are the top co-authors by number of shared publications:
- Thomas L. Griffiths
Princeton University
6 shared publications - Tal Linzen
Department of Linguistics, New York University
2 shared publications Aditya Yedetore
Boston University
1 shared publicationAsli Celikyilmaz
1 shared publicationDan Friedman
Department of Computer Science, Princeton University, Princeton, NJ 08542
1 shared publicationGianluca Bencomo
Department of Computer Science Princeton University
1 shared publication
Latest Publications
- 1 July 2026
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 49, e214
Not-so-strange love: Language models and generative linguistic theories are more compatible than they appear
Futrell and Mahowald frame the success of neural language models (LMs) as supporting gradient, usage-based linguistic theories. I argue that LMs can also instantiate theories based on formal structures – the types of theories seen in the generative tradition. This argument expands the space of theories that can be tested with LMs, potentially enabling reconciliations between usage-based and generative accounts.
- 1 July 2026
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 49, e214
Not-so-strange love: Language models and generative linguistic theories are more compatible than they appear
Futrell and Mahowald frame the success of neural language models (LMs) as supporting gradient, usage-based linguistic theories. I argue that LMs can also instantiate theories based on formal structures – the types of theories seen in the generative tradition. This argument expands the space of theories that can be tested with LMs, potentially enabling reconciliations between usage-based and generative accounts.
- Top Cited PapersOpen Access
- 20 May 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-14
Modeling rapid language learning by distilling Bayesian priors into artificial neural networks
Humans can learn languages from remarkably little experience. Developing computational models that explain this ability has been a major challenge in cognitive science. Existing approaches have been successful at explaining how humans generalize rapidly in controlled settings but are usually too restrictive to tractably handle naturalistic data. We show that learning from limited naturalistic data is possible with an approach that bridges the divide between two popular modeling traditions: Bayesian models and neural networks. This approach distills a Bayesian model’s inductive biases—the factors that guide generalization—into a neural network that has flexible representations. Like a Bayesian model, the resulting system can learn formal linguistic patterns from limited data. Like a neural network, it can also learn aspects of English syntax from naturally-occurring sentences. Thus, this model provides a single system that can learn rapidly and can handle naturalistic data.
- 20 May 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-14
Top Cited PapersOpen AccessModeling rapid language learning by distilling Bayesian priors into artificial neural networks
Humans can learn languages from remarkably little experience. Developing computational models that explain this ability has been a major challenge in cognitive science. Existing approaches have been successful at explaining how humans generalize rapidly in controlled settings but are usually too restrictive to tractably handle naturalistic data. We show that learning from limited naturalistic data is possible with an approach that bridges the divide between two popular modeling traditions: Bayesian models and neural networks. This approach distills a Bayesian model’s inductive biases—the factors that guide generalization—into a neural network that has flexible representations. Like a Bayesian model, the resulting system can learn formal linguistic patterns from limited data. Like a neural network, it can also learn aspects of English syntax from naturally-occurring sentences. Thus, this model provides a single system that can learn rapidly and can handle naturalistic data.
- Open Access
Identifying and Mitigating the Influence of the Prior Distribution in Large Language Models
Large language models (LLMs) sometimes fail to respond appropriately to deterministic tasks -- such as counting or forming acronyms -- because the implicit prior distribution they have learned over sequences of tokens influences their responses. In this work, we show that, in at least some cases, LLMs actually compute the information needed to perform these tasks correctly, and we identify some interventions that can allow them to access this information to improve their performance. First, we show that simply prompting the language model to not rely on its prior knowledge leads to dramatic improvements in prior-dominated tasks. We then use mechanistic interpretability techniques to localize the prior within the LLM and manipulate the extent to which that prior influences its responses. Specifically, we show that it is possible to identify layers of the underlying neural network that correlate with the prior probability of a response and that lightweight finetuning of these layers with basic prompts on prior-dominated tasks achieves high performance on held-out answers. These results suggest that the information required to produce a correct response is contained within the representations of the problems formed by the models. Furthermore, we show that this finetuning is significantly more effective for prior-dominated tasks, and that the error after finetuning is no longer correlated with the prior. Our results suggest that it may be possible to define effective methods for manipulating the extent to which LLMs rely upon their priors in solving problems, potentially increasing their performance in settings where LLMs hallucinate for reasons related to the prior probability of token sequences.
Open AccessIdentifying and Mitigating the Influence of the Prior Distribution in Large Language Models
Large language models (LLMs) sometimes fail to respond appropriately to deterministic tasks -- such as counting or forming acronyms -- because the implicit prior distribution they have learned over sequences of tokens influences their responses. In this work, we show that, in at least some cases, LLMs actually compute the information needed to perform these tasks correctly, and we identify some interventions that can allow them to access this information to improve their performance. First, we show that simply prompting the language model to not rely on its prior knowledge leads to dramatic improvements in prior-dominated tasks. We then use mechanistic interpretability techniques to localize the prior within the LLM and manipulate the extent to which that prior influences its responses. Specifically, we show that it is possible to identify layers of the underlying neural network that correlate with the prior probability of a response and that lightweight finetuning of these layers with basic prompts on prior-dominated tasks achieves high performance on held-out answers. These results suggest that the information required to produce a correct response is contained within the representations of the problems formed by the models. Furthermore, we show that this finetuning is significantly more effective for prior-dominated tasks, and that the error after finetuning is no longer correlated with the prior. Our results suggest that it may be possible to define effective methods for manipulating the extent to which LLMs rely upon their priors in solving problems, potentially increasing their performance in settings where LLMs hallucinate for reasons related to the prior probability of token sequences.
- Open Access
- 18 March 2025
- Published by Center for Open Science in PsyArXiv
Neural Networks Can Capture Human Concept Learning when Trained with Symbolic Priors
People can learn new concepts from a small number of examples by drawing on inductive biases that favor some hypotheses over others. These inductive biases have previously been captured by Bayesian models that use symbolic representations such as logical formulas to define prior distributions over hypotheses. We show that it is possible to create an artificial neural network that displays the same inductive biases as symbolic Bayesian models. Our approach is based on distilling a prior distribution from a symbolic Bayesian model into a neural network via meta-learning, a method for extracting the common structure from a set of tasks. This approach is used to create an artificial neural network with an inductive bias towards concepts expressed as short logical formulas. Analyzing results from previous behavioral experiments in which people learned logical concepts from a few examples shows that neural networks trained via meta-learning are able to capture human performance. These results demonstrate that it is possible to create artificial neural networks that capture types of inductive biases that have traditionally been expressed using symbolic models, providing a way to link accounts of human cognition at different levels of analysis.
- 18 March 2025
- Published by Center for Open Science in PsyArXiv
Open AccessNeural Networks Can Capture Human Concept Learning when Trained with Symbolic Priors
People can learn new concepts from a small number of examples by drawing on inductive biases that favor some hypotheses over others. These inductive biases have previously been captured by Bayesian models that use symbolic representations such as logical formulas to define prior distributions over hypotheses. We show that it is possible to create an artificial neural network that displays the same inductive biases as symbolic Bayesian models. Our approach is based on distilling a prior distribution from a symbolic Bayesian model into a neural network via meta-learning, a method for extracting the common structure from a set of tasks. This approach is used to create an artificial neural network with an inductive bias towards concepts expressed as short logical formulas. Analyzing results from previous behavioral experiments in which people learned logical concepts from a few examples shows that neural networks trained via meta-learning are able to capture human performance. These results demonstrate that it is possible to create artificial neural networks that capture types of inductive biases that have traditionally been expressed using symbolic models, providing a way to link accounts of human cognition at different levels of analysis.
- Open Access
Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks
Artificial neural networks can acquire many aspects of human knowledge from data, making them promising as models of human learning. But what those networks can learn depends upon their inductive biases -- the factors other than the data that influence the solutions they discover -- and the inductive biases of neural networks remain poorly understood, limiting our ability to draw conclusions about human learning from the performance of these systems. Cognitive scientists and machine learning researchers often focus on the architecture of a neural network as a source of inductive bias. In this paper we explore the impact of another source of inductive bias -- the initial weights of the network -- using meta-learning as a tool for finding initial weights that are adapted for specific problems. We evaluate four widely-used architectures -- MLPs, CNNs, LSTMs, and Transformers -- by meta-training 430 different models across three tasks requiring different biases and forms of generalization. We find that meta-learning can substantially reduce or entirely eliminate performance differences across architectures and data representations, suggesting that these factors may be less important as sources of inductive bias than is typically assumed. When differences are present, architectures and data representations that perform well without meta-learning tend to meta-train more effectively. Moreover, all architectures generalize poorly on problems that are far from their meta-training experience, underscoring the need for stronger inductive biases for robust generalization.
Open AccessTeasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks
Artificial neural networks can acquire many aspects of human knowledge from data, making them promising as models of human learning. But what those networks can learn depends upon their inductive biases -- the factors other than the data that influence the solutions they discover -- and the inductive biases of neural networks remain poorly understood, limiting our ability to draw conclusions about human learning from the performance of these systems. Cognitive scientists and machine learning researchers often focus on the architecture of a neural network as a source of inductive bias. In this paper we explore the impact of another source of inductive bias -- the initial weights of the network -- using meta-learning as a tool for finding initial weights that are adapted for specific problems. We evaluate four widely-used architectures -- MLPs, CNNs, LSTMs, and Transformers -- by meta-training 430 different models across three tasks requiring different biases and forms of generalization. We find that meta-learning can substantially reduce or entirely eliminate performance differences across architectures and data representations, suggesting that these factors may be less important as sources of inductive bias than is typically assumed. When differences are present, architectures and data representations that perform well without meta-learning tend to meta-train more effectively. Moreover, all architectures generalize poorly on problems that are far from their meta-training experience, underscoring the need for stronger inductive biases for robust generalization.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 14834-14847
Evaluating distillation methods for data-efficient syntax learning
Takateru Yamakoshi, Thomas L. Griffiths, R. Thomas McCoy, Robert D. Hawkins. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 14834-14847
Open AccessEvaluating distillation methods for data-efficient syntax learning
Takateru Yamakoshi, Thomas L. Griffiths, R. Thomas McCoy, Robert D. Hawkins. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- Open Access
Minimization of Boolean Complexity in In-Context Concept Learning
What factors contribute to the relative success and corresponding difficulties of in-context learning for Large Language Models (LLMs)? Drawing on insights from the literature on human concept learning, we test LLMs on carefully designed concept learning tasks, and show that task performance highly correlates with the Boolean complexity of the concept. This suggests that in-context learning exhibits a learning bias for simplicity in a way similar to humans.
Open AccessMinimization of Boolean Complexity in In-Context Concept Learning
What factors contribute to the relative success and corresponding difficulties of in-context learning for Large Language Models (LLMs)? Drawing on insights from the literature on human concept learning, we test LLMs on carefully designed concept learning tasks, and show that task performance highly correlates with the Boolean complexity of the concept. This suggests that in-context learning exhibits a learning bias for simplicity in a way similar to humans.
- Top Cited PapersOpen Access
- 4 October 2024
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 121 (41)
Embers of autoregression show how large language models are shaped by the problem they are trained to solve
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach—which we call the teleological approach—we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4’s accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system—one that has been shaped by its own particular set of pressures.
- 4 October 2024
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 121 (41)
Top Cited PapersOpen AccessEmbers of autoregression show how large language models are shaped by the problem they are trained to solve
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach—which we call the teleological approach—we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4’s accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system—one that has been shaped by its own particular set of pressures.
- 23 September 2024
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 47, e155
Meta-learning as a bridge between neural networks and symbolic Bayesian models
Meta-learning is even more broadly relevant to the study of inductive biases than Binz et al. suggest: Its implications go beyond the extensions to rational analysis that they discuss. One noteworthy example is that meta-learning can act as a bridge between the vector representations of neural networks and the symbolic hypothesis spaces used in many Bayesian models.
- 23 September 2024
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 47, e155
Meta-learning as a bridge between neural networks and symbolic Bayesian models
Meta-learning is even more broadly relevant to the study of inductive biases than Binz et al. suggest: Its implications go beyond the extensions to rational analysis that they discuss. One noteworthy example is that meta-learning can act as a bridge between the vector representations of neural networks and the symbolic hypothesis spaces used in many Bayesian models.
- Open Access
Distilling Symbolic Priors for Concept Learning into Neural Networks
Humans can learn new concepts from a small number of examples by drawing on their inductive biases. These inductive biases have previously been captured by using Bayesian models defined over symbolic hypothesis spaces. Is it possible to create a neural network that displays the same inductive biases? We show that inductive biases that enable rapid concept learning can be instantiated in artificial neural networks by distilling a prior distribution from a symbolic Bayesian model via meta-learning, an approach for extracting the common structure from a set of tasks. By generating the set of tasks used in meta-learning from the prior distribution of a Bayesian model, we are able to transfer that prior into a neural network. We use this approach to create a neural network with an inductive bias towards concepts expressed as short logical formulas. Analyzing results from previous behavioral experiments in which people learned logical concepts from a few examples, we find that our meta-trained models are highly aligned with human performance.
Open AccessDistilling Symbolic Priors for Concept Learning into Neural Networks
Humans can learn new concepts from a small number of examples by drawing on their inductive biases. These inductive biases have previously been captured by using Bayesian models defined over symbolic hypothesis spaces. Is it possible to create a neural network that displays the same inductive biases? We show that inductive biases that enable rapid concept learning can be instantiated in artificial neural networks by distilling a prior distribution from a symbolic Bayesian model via meta-learning, an approach for extracting the common structure from a set of tasks. By generating the set of tasks used in meta-learning from the prior distribution of a Bayesian model, we are able to transfer that prior into a neural network. We use this approach to create a neural network with an inductive bias towards concepts expressed as short logical formulas. Analyzing results from previous behavioral experiments in which people learned logical concepts from a few examples, we find that our meta-trained models are highly aligned with human performance.