LastDance: a Diagnostic-Driven Autoresearch Framework for Retrieval-Augmented Generation with Compact Language Models

Authors:
Preprint
Publisher website
Google Scholar
Abstract
We present LastDance, an autoresearch framework that autonomously discovers optimized Retrieval-Augmented Generation pipelines for compact language models under token budget constraints. The framework guides a coding agent (Claude Code paired with Qwen3-Coder-Next as the underlying coder model) through a proposed project structure comprising three instruction files, a modifiable pipeline file, a fixed evaluation script with diagnostic report generation, and a curated research knowledge base. After each experiment, a diagnostic module classifies every evaluation sample into four failure categories and recommends which pipeline stage to modify next. This helps the agent make research decisions based on diagnostic evidence. The methodology discovered by LastDance, KnapsackRAG, formulates passage selection as a 0/1 integer linear program with cardinality, coverage, and pairwise redundancy constraints. On the NQ Open and TriviaQA validation sets, KnapsackRAG achieves 36.9% and 36.2% relative F1 improvements over the strongest top-k baseline with Qwen2.5-1.5B-Instruct as the generator. The discovered configuration transfers to Llama-3.2-3B-Instruct without modification, yielding 99.3% and 17.3% relative F1 gains on the same two datasets.

This publication has 11 references indexed in Scilit: