SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Open Access
- 1 January 2024
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 12622-12645
Abstract
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Data Interpreter: An LLM Agent for Data Science2025
- Computational reproducibility of Jupyter notebooks from biomedical publicationsGigaScience, 2024
- TaskBench: Benchmarking Large Language Models for Task Automation2024
- DebugBench: Evaluating Debugging Capability of Large Language Models2024
- MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization2024
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code GenerationIEEE Transactions on Software Engineering, 2023
- Measuring and Narrowing the Compositionality Gap in Language Models2023
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation2023
- NLP Reproducibility For All: Understanding Experiences of Beginners2023
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019