Richong Zhang

Dense retrieval usually fails in two ways: ranking non-relevant documents too high, or ranking relevant documents too low. We focus on the second case from the query side: documents that are genuinely relevant but receive low retrieval scores under certain query formulations, forming hard-positive query–document pairs. To study this failure mode systematically, we build on recent token-alignment work, which analyzes vocabulary logits obtained by projecting retriever representations through an architecture-specific token prediction head and shows that low overlap among top-ranked logit tokens can cause relevant documents to be scored low. This suggests a practical route to hard positives: generate relevant queries that rely on document phrases that receive low ranks under these logits. We therefore define model-specific Token and Phrase Representativeness Scores (TRS/PRS) to discover tokens and key phrases that appear in a document but are poorly expressed by its embedding. Using high-PRS phrases as anchors, we automatically construct challenging yet relevant queries, yielding hard-positive query–document pairs. Experiments across multiple retrievers and datasets show that PRS-anchored queries induce substantially larger drops for dense retrievers than other query construction strategies, revealing a dense-specific weakness. Moreover, when mixed with standard hard negatives during contrastive fine-tuning, these hard positives improve retrieval on our constructed hard-positive benchmark and can also improve standard in-domain retrieval benchmarks. Our code is publicly available at https://github.com/wzy2001wzy/HardPositive.

Retrieval-Augmented Generation (RAG) often struggles with knowledge conflicts, where model-internal parametric knowledge overrides retrieved evidence, leading to unfaithful outputs. Existing approaches are often limited, relying either on superficial decoding adjustments or weight editing that necessitates ground-truth targets. Through layer-wise analysis, we attribute this failure to a parametric suppression phenomenon: specifically, in deep layers, certain FFN layers overwrite context-sensitive representations with memorized priors. To address this, we propose CoRect (Context-Aware Logit Contrast for Hidden State Rectification). By contrasting logits from contextualized and non-contextualized forward passes, CoRect identifies layers that exhibit high parametric bias without requiring ground-truth labels. It then rectifies the hidden states to preserve evidence-grounded information. Across question answering (QA) and summarization benchmarks, CoRect consistently improves faithfulness and reduces hallucinations compared to strong baselines.