Charles L. A. Clarke

LLM-based query reformulation can improve retrieval, but no single reformulation strategy is consistently optimal across queries, domains, retrievers, or model backbones. This creates an inference-time decision problem: ''Given an original query and a pool of candidate reformulations, which one should be issued to the retriever?''. Existing studies are hard to compare because they use different reformulator pools, retrievers, relevance signals, training labels, and evaluation metrics. We introduce QueryRoute, a benchmark that freezes the expensive artifacts needed to study this decision reproducibly: original queries, generated variants, ranked lists under multiple retrievers, retrieval scores, and per-query oracle labels. The benchmark contains 3,757 queries, 11 candidate systems, five reformulator backbones, and three retrievers across TREC DL, BEIR, and BRIGHT, yielding 619,905 retrieval outcomes. We benchmark supervised classification, routing, QPP, and LLM-as-judge selectors. Results show substantial oracle headroom over fixed reformulators, but current selectors recover only part of it; selector rankings change across retrievers, and similar mean effectiveness can hide different query-level behavior. The released artifacts and evaluation harness allow future selectors to be compared without regenerating variants, rerunning retrieval, or rebuilding judge pipelines. Code and data are available at https://github.com/haisonle001/QueryRoute.

Large language models (LLMs) are increasingly used to produce the relevance labels that underpin the evaluation of information retrieval (IR) and retrieval-augmented generation systems. Yet delegating relevance assessment to an LLM raises a meta-evaluation question that is hard to answer with existing resources: when, and how much, can we trust an automatic judge? We present LLMJudge, a resource for meta-evaluating LLM-generated relevance judgments. LLMJudge collects 42 complete judgment sets over the TREC 2023 Deep Learning track, spanning fine-tuned, proprietary, and open-source models and a wide range of prompting strategies. Each judgment set is paired with official human qrels, enabling two complementary axes of analysis: label agreement with human assessors (e.g., Cohen's κ, Krippendorff's α) and system-ranking fidelity (e.g., Kendall's τ, Spearman's ρ). Beyond the data, we release a standardized format, baselines, and an evaluation harness so that new judges can be plugged in and compared reproducibly. Our analysis shows that high label agreement is neither necessary nor sufficient for faithful system ranking, that open-source judges are markedly more stable than proprietary ones, and that fine-tuning improves correlation at the cost of reliability. LLMJudge is publicly available and provides a reusable testbed for auditing the judges who increasingly judge our systems. Full resources, data, and detailed results are available at https://llm4eval.github.io/LLMJudge-benchmark/.

Agents in knowledge-intensive domains require not only knowledge retrieval, but also the skills to effectively use retrieved knowledge. Since domain knowledge is often highly complex, agents must possess strong reasoning capabilities in order to perform tasks effectively. In the legal domain, different legal systems exhibit substantial differences, and the reasoning logic for applying legal knowledge varies significantly across systems. Meanwhile, existing skill collections lack a systematic skill framework grounded in domain knowledge, limiting their scalability and extensibility. To address these challenges, we present LexSkill, a reasoning-oriented legal skill library for the Chinese legal system. Within this framework, legal experts construct 36 atomic skills, further organized into 180 sub-atomic skills and 2 compound skills. Since legal problem solving also requires grounded access to legal resources and coordinated reasoning across multiple steps, we further introduce Chinese-Law-Harness, which augments the skill library with MCP modules built over a case repository and a statutory repository, and incorporates a multi-agent system for skill orchestration and execution. Evaluations on existing legal AI benchmarks show that leveraging the skill library through the proposed harness significantly outperforms direct problem solving without structured support, verifying the effectiveness of our skills and harness.

Recent research has shown that neural information retrieval techniques may be susceptible to adversarial attacks. Adversarial attacks seek to manipulate the ranking of documents, with the intention of exposing users to targeted content. In this article, we introduce the Embedding Perturbation Rank Attack ( EMPRA ) method, a novel approach designed to perform adversarial attacks on black-box Neural Ranking Models (NRMs) . EMPRA manipulates sentence-level embeddings, guiding them towards pertinent context related to the query while preserving semantic integrity. This process generates adversarial texts that seamlessly integrate with the original content and remain imperceptible to humans. Our extensive evaluation conducted on the widely used MS MARCO V1 passage collection as well as the TREC DL 2019 and TREC DL 2020 benchmarks, demonstrate the effectiveness of EMPRA against a wide range of state-of-the-art baselines in promoting a specific set of target documents within a given ranked results. Specifically, on MS MARCO Dev set queries, EMPRA successfully achieves a re-ranking of almost 96% of target documents originally ranked between 51 and 100 to rank within the top 10. Furthermore, EMPRA does not rely on surrogate models for generating adversarial documents, enhancing its robustness against various victim NRMs in realistic settings.