Alexis Chevalier
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Alexis Chevalier has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Anirudh Ajith
Department of Computer Science & Princeton Language and Intelligence Princeton University
3 shared publications - Alexander Wettig
Princeton Language and Intelligence Princeton University
2 shared publications - Danqi Chen
Princeton Language and Intelligence (PLI) Princeton University
2 shared publications - Danqi Chen
Princeton Language and Intelligence (PLI) Princeton University
1 shared publication - Mengzhou Xia
Princeton Language and Intelligence (PLI) Princeton University
1 shared publication - Tanya Goyal
Princeton Language and Intelligence (PLI) Princeton University
1 shared publication
Latest Publications
- Top Cited PapersOpen Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 15068-15083
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 15068-15083
Top Cited PapersOpen AccessLitSearch: A Retrieval Benchmark for Scientific Literature Search
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
- Open Access
Adapting Language Models to Compress Contexts
Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These language models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments, and summary vectors from all previous segments are used in language modeling. We fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompressors on in-context learning by compressing task demonstrations and find that summary vectors are good substitutes for plain-text demonstrations, increasing accuracy while reducing inference costs. Finally, we explore the benefits of pre-computing summary vectors for large corpora by applying summary vectors to retrievalaugmented language modeling and a passage re-ranking task. Overall, AutoCompressors emerge as a simple and inexpensive solution to extend the context window of LMs while speeding up inference over long contexts.
Open AccessAdapting Language Models to Compress Contexts
Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These language models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments, and summary vectors from all previous segments are used in language modeling. We fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompressors on in-context learning by compressing task demonstrations and find that summary vectors are good substitutes for plain-text demonstrations, increasing accuracy while reducing inference costs. Finally, we explore the benefits of pre-computing summary vectors for large corpora by applying summary vectors to retrievalaugmented language modeling and a passage re-ranking task. Overall, AutoCompressors emerge as a simple and inexpensive solution to extend the context window of LMs while speeding up inference over long contexts.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 3829-3846
Adapting Language Models to Compress Contexts
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 3829-3846
Open AccessAdapting Language Models to Compress Contexts
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.