Maik Fröbe

The lack of efficiency in neural information retrieval remains one of the primary obstacles to deploying neural retrieval models as a first-stage retriever at scale. While recent tools have improved the standardized measurement of model efficiency, substantial progress is still needed to enable systematic comparative evaluation, for example, in terms of standards for systems and hardware configurations, cloud-based evaluation, benchmarks, and reproducibility. Beyond measurement, the IR community also needs stronger incentives to move in this direction, such as cost-efficiency as a review criterion or as efficiency and/or effectiveness measures in shared tasks, related teaching materials, efficiency-oriented user studies, and specialized awards for efficiency achievements. In particular, developing more efficient variants of highly effective retrieval algorithms should become an admissible research goal for PhD students if cost-efficiency is to become a first-class design objective in~IR. With ReNeuIR, we have established a recurring forum where these questions and new ideas are discussed and where the community comes together to collaboratively evaluate and improve efficiency benchmarking frameworks---most notably through the organization of a shared task focused on efficiency and reproducibility.