AAAnirudh Ajith

Anirudh Ajith

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Anirudh Ajith has a total of 12 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Downstream Trade-offs of a Family of Text Watermarks

    by Anirudh Ajith, Sameer Singh, Danish Pruthi

    Anirudh Ajith, Sameer Singh, Danish Pruthi. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.


  • Top Cited Papers
    Open Access

    LitSearch: A Retrieval Benchmark for Scientific Literature Search

    by Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao

    Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.


  • Open Access

    Adapting Language Models to Compress Contexts

    by Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen

    Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These language models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments, and summary vectors from all previous segments are used in language modeling. We fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompressors on in-context learning by compressing task demonstrations and find that summary vectors are good substitutes for plain-text demonstrations, increasing accuracy while reducing inference costs. Finally, we explore the benefits of pre-computing summary vectors for large corpora by applying summary vectors to retrievalaugmented language modeling and a passage re-ranking task. Overall, AutoCompressors emerge as a simple and inexpensive solution to extend the context window of LMs while speeding up inference over long contexts.


  • Open Access

    InstructEval: Systematic Evaluation of Instruction Selection Methods

    by Anirudh Ajith, Chris Pan, Mengzhou Xia, Ameet Deshpande, Karthik Narasimhan

    In-context learning (ICL) performs tasks by prompting a large language model (LLM) using an instruction and a small set of annotated examples called demonstrations. Recent work has shown that precise details of the inputs used in the ICL prompt significantly impact performance, which has incentivized instruction selection algorithms. The effect of instruction-choice however is severely underexplored, with existing analyses restricted to shallow subsets of models and tasks, limiting the generalizability of their insights. We develop InstructEval, an ICL evaluation suite to conduct a thorough assessment of these techniques. The suite includes 13 open-sourced LLMs of varying scales from four model families, and covers nine tasks across three categories. Using the suite, we evaluate the relative performance of seven popular instruction selection methods over five metrics relevant to ICL. Our experiments reveal that using curated manually-written instructions or simple instructions without any task-specific descriptions often elicits superior ICL performance overall than that of automatic instruction-induction methods, pointing to a lack of generalizability among the latter. We release our evaluation suite for benchmarking instruction selection approaches and enabling more generalizable methods in this space.

  • Open Access

    Adapting Language Models to Compress Contexts

    by Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.