RFRobert Frank

Robert Frank

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Robert Frank has a total of 47 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Brain and grammar: revealing electrophysiological basic structures with competing statistical models

    by Andrea Cometa, Chiara Battaglini, Fiorenzo Artoni, Matteo Greco, Robert Frank, Claudia Repetto, Franco Bottoni, Stefano F Cappa, Silvestro Micera, Emiliano Ricciardi, et al.

    Acoustic, lexical and syntactic information is simultaneously processed in the brain. Therefore, distinguishing the electrophysiological activity pertaining to these components requires complex and indirect strategies. Capitalizing on previous works which factor out acoustic information, we could concentrate on the lexical and syntactic contribution to language processing by testing competing statistical models. We exploited EEG recordings and compared different surprisal models selectively involving lexical information, part of speech or syntactic structures in various combinations. EEG responses were recorded in 32 participants during listening to affirmative active declarative sentences and compared the activation corresponding to basic syntactic structures, such as noun phrases vs verb phrases. Lexical and syntactic processing activates different frequency bands, different time windows and different networks. Moreover, surprisal models based on part of speech inventory only do not explain well the electrophysiological data, while those including syntactic information do. Finally, we confirm previous measures obtained with intracortical recordings independently supporting the original hypothesis addressed here in a robust way.

  • Open Access

    LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition

    by Xiaomeng Zhu, Robert Frank

    Xiaomeng Zhu, Robert Frank. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

  • Top Cited Papers
    Open Access

    How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure

    by Michael Wilson, Jackson Petty, Robert Frank

    Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.1


  • Open Access

    How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure

    by Michael Wilson, Jackson Petty, Robert Frank

    Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained Transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.s reported here are available at https://github.com/clay-lab/structural-alternations.

  • Attention and Locality: On Clause-Boundedness and Its Exceptions in Multiple Sluicing

    by Matthew Barros, Robert Frank

    We provide an account of clause-boundedness in multiple sluicing that also captures its exceptions. Clause-boundedness arises whenever an embedded clause’s subject is not coreferential with a topical discourse referent in the embedding clause. Our account ties clause-boundedness to discourse factors. We discuss implementations that import sensitivity to information structure into the syntax, and compare our approach with recent work—in particular, Grano and Lasnik 2018 and “short source” accounts (most recently, Abels and Dayal 2017, 2021)—and demonstrate that these accounts both under- and overgenerate. The empirical coverage of our account argues against purely syntacticized agreement-based approaches to clause-boundedness.


  • Open Access

    Inductive Bias Is in the Eye of the Beholder

    by Michael Wilson, Robert Frank

    Michael Wilson, Robert Frank. Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. 2023.


  • Top Cited Papers
    Open Access

    How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech

    by Aditya Yedetore, Tal Linzen, Robert Frank, R. Thomas McCoy

    Aditya Yedetore, Tal Linzen, Robert Frank, R. Thomas McCoy. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.


  • Open Access

    Do Language Models Learn Position-Role Mappings?

    by Jackson Petty, Michael Wilson, Robert Frank

    How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained language models (BERT, RoBERTa, and DistilBERT) exhibit knowledge of these mappings, and whether this knowledge persists across alternations in syntactic, structural, and lexical alternations. In Experiment 1, we show that these neural models do indeed recognize distinctions between theme and recipient roles in ditransitive constructions, and that these distinct patterns are shared across construction type. We strengthen this finding in Experiment 2 by showing that fine-tuning these language models on novel theme- and recipient-like tokens in one paradigm allows the models to make correct predictions about their placement in other paradigms, suggesting that the knowledge of these mappings is shared rather than independently learned. We do, however, observe some limitations of this generalization when tasks involve constructions with novel ditransitive verbs, hinting at a degree of lexical specificity which underlies model performance.

  • Open Access

    Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora

    by Robert Frank, Jackson Petty

    Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables. Past work has raised doubts about the ability of recurrent networks to meet this challenge. In this paper, we explore this question in the context of a fragment of English that incorporates the relevant sort of contextual variability. We consider sequence-to-sequence architectures with recurrent units and show that such networks are capable of learning semantic interpretations for reflexive anaphora which generalize to novel antecedents. We explore the effect of attention mechanisms and different recurrent unit types on the type of training data that is needed for success as measured in two ways: how much lexical support is needed to induce an abstract reflexive meaning (i.e., how many distinct reflexive antecedents must occur during training) and what contexts must a noun phrase occur in to support generalization of reflexive interpretation to this noun phrase?

  • Open Access

    Finding Hierarchical Structure in Neural Stacks Using Unsupervised Parsing

    by William Merrill, Lenny Khazan, Noah Amsel, Yiding Hao, Simon Mendelsohn, Robert Frank