Robert Frank
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Robert Frank has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
Noah Amsel
3 shared publicationsSimon Mendelsohn
3 shared publicationsWilliam Merrill
3 shared publicationsYiding Hao
3 shared publications- Jackson Petty
Department of Linguistics New York University
2 shared publications - Michael Wilson
Yale University, USA
2 shared publications
Latest Publications
- Open Access
Brain and grammar: revealing electrophysiological basic structures with competing statistical models
Acoustic, lexical and syntactic information is simultaneously processed in the brain. Therefore, distinguishing the electrophysiological activity pertaining to these components requires complex and indirect strategies. Capitalizing on previous works which factor out acoustic information, we could concentrate on the lexical and syntactic contribution to language processing by testing competing statistical models. We exploited EEG recordings and compared different surprisal models selectively involving lexical information, part of speech or syntactic structures in various combinations. EEG responses were recorded in 32 participants during listening to affirmative active declarative sentences and compared the activation corresponding to basic syntactic structures, such as noun phrases vs verb phrases. Lexical and syntactic processing activates different frequency bands, different time windows and different networks. Moreover, surprisal models based on part of speech inventory only do not explain well the electrophysiological data, while those including syntactic information do. Finally, we confirm previous measures obtained with intracortical recordings independently supporting the original hypothesis addressed here in a robust way.
Open AccessBrain and grammar: revealing electrophysiological basic structures with competing statistical models
Acoustic, lexical and syntactic information is simultaneously processed in the brain. Therefore, distinguishing the electrophysiological activity pertaining to these components requires complex and indirect strategies. Capitalizing on previous works which factor out acoustic information, we could concentrate on the lexical and syntactic contribution to language processing by testing competing statistical models. We exploited EEG recordings and compared different surprisal models selectively involving lexical information, part of speech or syntactic structures in various combinations. EEG responses were recorded in 32 participants during listening to affirmative active declarative sentences and compared the activation corresponding to basic syntactic structures, such as noun phrases vs verb phrases. Lexical and syntactic processing activates different frequency bands, different time windows and different networks. Moreover, surprisal models based on part of speech inventory only do not explain well the electrophysiological data, while those including syntactic information do. Finally, we confirm previous measures obtained with intracortical recordings independently supporting the original hypothesis addressed here in a robust way.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 13835-13850
LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition
Xiaomeng Zhu, Robert Frank. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 13835-13850
Open AccessLIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition
Xiaomeng Zhu, Robert Frank. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- Top Cited PapersOpen Access
- 13 November 2023
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 11, 1377-1395
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.1
- 13 November 2023
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 11, 1377-1395
Top Cited PapersOpen AccessHow Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.1
- Open Access
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained Transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.s reported here are available at https://github.com/clay-lab/structural-alternations.
Open AccessHow Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained Transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.s reported here are available at https://github.com/clay-lab/structural-alternations.
- 23 June 2023
- Published by MIT Press in Linguistic Inquiry
- Vol. 54 (4) , 649-684
Attention and Locality: On Clause-Boundedness and Its Exceptions in Multiple Sluicing
We provide an account of clause-boundedness in multiple sluicing that also captures its exceptions. Clause-boundedness arises whenever an embedded clause’s subject is not coreferential with a topical discourse referent in the embedding clause. Our account ties clause-boundedness to discourse factors. We discuss implementations that import sensitivity to information structure into the syntax, and compare our approach with recent work—in particular, Grano and Lasnik 2018 and “short source” accounts (most recently, Abels and Dayal 2017, 2021)—and demonstrate that these accounts both under- and overgenerate. The empirical coverage of our account argues against purely syntacticized agreement-based approaches to clause-boundedness.
- 23 June 2023
- Published by MIT Press in Linguistic Inquiry
- Vol. 54 (4) , 649-684
Attention and Locality: On Clause-Boundedness and Its Exceptions in Multiple Sluicing
We provide an account of clause-boundedness in multiple sluicing that also captures its exceptions. Clause-boundedness arises whenever an embedded clause’s subject is not coreferential with a topical discourse referent in the embedding clause. Our account ties clause-boundedness to discourse factors. We discuss implementations that import sensitivity to information structure into the syntax, and compare our approach with recent work—in particular, Grano and Lasnik 2018 and “short source” accounts (most recently, Abels and Dayal 2017, 2021)—and demonstrate that these accounts both under- and overgenerate. The empirical coverage of our account argues against purely syntacticized agreement-based approaches to clause-boundedness.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 152-162
Inductive Bias Is in the Eye of the Beholder
Michael Wilson, Robert Frank. Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 152-162
Open AccessInductive Bias Is in the Eye of the Beholder
Michael Wilson, Robert Frank. Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. 2023.
- Top Cited PapersOpen Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 9370-9393
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech
Aditya Yedetore, Tal Linzen, Robert Frank, R. Thomas McCoy. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 9370-9393
Top Cited PapersOpen AccessHow poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech
Aditya Yedetore, Tal Linzen, Robert Frank, R. Thomas McCoy. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
- Open Access
Do Language Models Learn Position-Role Mappings?
How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained language models (BERT, RoBERTa, and DistilBERT) exhibit knowledge of these mappings, and whether this knowledge persists across alternations in syntactic, structural, and lexical alternations. In Experiment 1, we show that these neural models do indeed recognize distinctions between theme and recipient roles in ditransitive constructions, and that these distinct patterns are shared across construction type. We strengthen this finding in Experiment 2 by showing that fine-tuning these language models on novel theme- and recipient-like tokens in one paradigm allows the models to make correct predictions about their placement in other paradigms, suggesting that the knowledge of these mappings is shared rather than independently learned. We do, however, observe some limitations of this generalization when tasks involve constructions with novel ditransitive verbs, hinting at a degree of lexical specificity which underlies model performance.
Open AccessDo Language Models Learn Position-Role Mappings?
How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained language models (BERT, RoBERTa, and DistilBERT) exhibit knowledge of these mappings, and whether this knowledge persists across alternations in syntactic, structural, and lexical alternations. In Experiment 1, we show that these neural models do indeed recognize distinctions between theme and recipient roles in ditransitive constructions, and that these distinct patterns are shared across construction type. We strengthen this finding in Experiment 2 by showing that fine-tuning these language models on novel theme- and recipient-like tokens in one paradigm allows the models to make correct predictions about their placement in other paradigms, suggesting that the knowledge of these mappings is shared rather than independently learned. We do, however, observe some limitations of this generalization when tasks involve constructions with novel ditransitive verbs, hinting at a degree of lexical specificity which underlies model performance.
- Open Access
Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora
Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables. Past work has raised doubts about the ability of recurrent networks to meet this challenge. In this paper, we explore this question in the context of a fragment of English that incorporates the relevant sort of contextual variability. We consider sequence-to-sequence architectures with recurrent units and show that such networks are capable of learning semantic interpretations for reflexive anaphora which generalize to novel antecedents. We explore the effect of attention mechanisms and different recurrent unit types on the type of training data that is needed for success as measured in two ways: how much lexical support is needed to induce an abstract reflexive meaning (i.e., how many distinct reflexive antecedents must occur during training) and what contexts must a noun phrase occur in to support generalization of reflexive interpretation to this noun phrase?
Open AccessSequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora
Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables. Past work has raised doubts about the ability of recurrent networks to meet this challenge. In this paper, we explore this question in the context of a fragment of English that incorporates the relevant sort of contextual variability. We consider sequence-to-sequence architectures with recurrent units and show that such networks are capable of learning semantic interpretations for reflexive anaphora which generalize to novel antecedents. We explore the effect of attention mechanisms and different recurrent unit types on the type of training data that is needed for success as measured in two ways: how much lexical support is needed to induce an abstract reflexive meaning (i.e., how many distinct reflexive antecedents must occur during training) and what contexts must a noun phrase occur in to support generalization of reflexive interpretation to this noun phrase?
- Open Access
- 1 January 2019
- Published by Association for Computational Linguistics (ACL)
Finding Hierarchical Structure in Neural Stacks Using Unsupervised Parsing
- 1 January 2019
- Published by Association for Computational Linguistics (ACL)
Open AccessFinding Hierarchical Structure in Neural Stacks Using Unsupervised Parsing