Emily Dinan
Research Subjects
Research Interests
Emily Dinan has a total of 31 co-authors. Here are the top co-authors by number of shared publications:
- Athul Paul Jacob
Meta AI, 1 Hacker Way, Menlo Park, CA, USA
2 shared publications - Mike Lewis
Meta AI
2 shared publications - Weiyan Shi
Department of Electrical and Computer Engineering and Khoury College of Computer Sciences, Northeastern University , Boston, MA 02115
2 shared publications - Adam Lerer
Facebook AI Research
1 shared publication - Adithya Renduchintala
Meta AI, 1 Hacker Way, Menlo Park, CA, USA
1 shared publication - Adithya Renduchintala1 shared publication
Latest Publications
- Open Access
Effective Theory of Transformers at Initialization
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.
Open AccessEffective Theory of Transformers at Initialization
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.
- Top Cited Papers
- 9 December 2022
- Published by American Association for the Advancement of Science (AAAS) in Science
- Vol. 378 (6624) , 1067-1074
Human-level play in the game ofDiplomacyby combining language models with strategic reasoning
Despite much progress in training artificial intelligence (AI) systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance inDiplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous onlineDiplomacyleague, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.
- 9 December 2022
- Published by American Association for the Advancement of Science (AAAS) in Science
- Vol. 378 (6624) , 1067-1074
Top Cited PapersHuman-level play in the game ofDiplomacyby combining language models with strategic reasoning
Despite much progress in training artificial intelligence (AI) systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance inDiplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous onlineDiplomacyleague, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.
- Open Access
AutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Replies
Existing approaches built separate classifiers to detect nonsense in dialogues. In this paper, we show that without external classifiers, dialogue models can detect errors in their own messages introspectively, by calculating the likelihood of replies that are indicative of poor messages. For example, if an agent believes its partner is likely to respond "I don't understand" to a candidate message, that message may not make sense, so an alternative message should be chosen. We evaluate our approach on a dataset from the game Diplomacy, which contains long dialogues richly grounded in the game state, on which existing models make many errors. We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy. We then design AutoReply, an algorithm to search for such discriminative replies automatically, given a small number of annotated dialogue examples. We find that AutoReply-generated replies outperform handcrafted replies and perform on par with carefully fine-tuned large supervised models. Results also show that one single reply without much computation overheads can also detect dialogue nonsense reasonably well.
Open AccessAutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Replies
Existing approaches built separate classifiers to detect nonsense in dialogues. In this paper, we show that without external classifiers, dialogue models can detect errors in their own messages introspectively, by calculating the likelihood of replies that are indicative of poor messages. For example, if an agent believes its partner is likely to respond "I don't understand" to a candidate message, that message may not make sense, so an alternative message should be chosen. We evaluate our approach on a dataset from the game Diplomacy, which contains long dialogues richly grounded in the game state, on which existing models make many errors. We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy. We then design AutoReply, an algorithm to search for such discriminative replies automatically, given a small number of annotated dialogue examples. We find that AutoReply-generated replies outperform handcrafted replies and perform on par with carefully fine-tuned large supervised models. Results also show that one single reply without much computation overheads can also detect dialogue nonsense reasonably well.