Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2407.02883

CoIR: A Comprehensive Benchmark for Code Information Retrieval Models

Xiangyang Li·Kuicai Dong·Yi Quan Lee·Wei Xia·Hao Zhang·Xinyi Dai·Yasheng Wang·Ruiming Tang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Despite the substantial success of Information Retrieval (IR) in various NLP tasks, most IR systems predominantly handle queries and corpora in natural language, neglecting the domain of code retrieval. Code retrieval is critically important yet remains under-explored, with existing methods and benchmarks inadequately representing the diversity of code in various domains and tasks. Addressing this gap, we present COIR (Code Information Retrieval Benchmark), a robust and comprehensive benchmark specifically designed to assess code retrieval capabilities. COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains. We first discuss the construction of COIR and its diverse dataset composition. Further, we evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems. To facilitate easy adoption and integration within existing research workflows, COIR has been developed as a user-friendly Python framework, readily installable via pip. It shares same data schema as other popular benchmarks like MTEB and BEIR, enabling seamless cross-benchmark evaluations. Through COIR, we aim to invigorate research in the code retrieval domain, providing a versatile benchmarking tool that encourages further development and exploration of code retrieval systems. https://github.com/CoIR-team/coir.

Explore related subjects

cs.IRcs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, Ruiming Tang. 2025-06-06. CoIR: A Comprehensive Benchmark for Code Information Retrieval Models. https://arxiv.org/abs/2407.02883

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

FlowRec: Prior-Informed Flow Matching for Efficient Sequential Recommendation Generation

Sequential recommendation aims to predict each user's next preferred item based on their historical interactions. Recently, diffusion-based approaches have demonstrated strong generative capability in modeling complex user preferences. However, they still face two inherent limitations: (i) Gaussian priors are misaligned with user-specific interests, and curved noise schedules lead to error accumulation and unstable training; (ii) the stochastic denoising process introduces additional randomness and substantial computational overhead. To address these issues, we propose FlowRec, a flow-matching-based framework that formulates preference evolution as continuous flows between personalized priors and target items. Specifically, FlowRec constructs an informative behavior-based prior distribution derived from users' historical interactions, offering a distributionally closer initialization to the target distribution. It then learns a vector field to guide straight preference flows toward target interests. Moreover, a single-step alignment objective with positive and negative samples further enhances semantic consistency between generated representations and ground-truth items. Finally, FlowRec adopts deterministic ODE-based generation, achieving efficient and stable inference. Extensive experiments on multiple benchmark datasets demonstrate that FlowRec consistently outperforms state-of-the-art baselines in both recommendation accuracy and inference efficiency.

cs.IR↗
arXiv2026

Semantic Candidate-Job Matching: A Comparative Evaluation of Dense Embedding Models in Hybrid Retrieval

This paper presents a comparative evaluation of dense embedding models for semantic candidate-job matching in high-volume staffing workflows. Incoming job descriptions are converted into structured English search text and language-specific keywords through LLM-based parsing, and candidate profiles are indexed as semantically enriched resume representations. We evaluate EmbeddingGemma (base) against EmbeddingGemma fine-tuned with Cached Multiple Negatives Ranking Loss (MNRL) within a unified hybrid retrieval pipeline that fuses vector similarity and full-text relevance via reciprocal rank fusion (RRF), and benchmark both against the MPNet model on a batch comparative evaluation dataset scored through the deployed job-candidate matching scoring pipeline. We further document, with mathematical detail, the broader set of contrastive fine-tuning objectives considered during model development (including AnglE/CoSENT-style refinement) and the empirical rationale for retaining Cached-MNRL-only adaptation as the preferred configuration. To support reproducible model selection, we define a broader evaluation framework comprising standard information retrieval metrics (Recall@K, mean reciprocal rank, nDCG) under the exact hybrid-retrieval protocol; the metrics used for the evaluation reported in this paper are fine-tuning convergence diagnostics and a batch comparative evaluation using the deployed AI-Match score and an independent LLM-as-a-Judge relevance score, and we state this scope explicitly rather than implying the full framework was measured. The paper addresses the gap between general-purpose embedding benchmarks and enterprise job-candidate matching constraints, providing a structured basis for comparing embedding strategies under realistic job-candidate retrieval conditions.

cs.IR↗
arXiv2026

From Ranked Documents to Reliable Contexts: An Answer-Oriented Context Construct Framework for AI Search

Traditional Web search follows a human-facing paradigm in which users inspect ranked documents and synthesize information themselves. In AI Search, retrieved documents instead serve as inputs to a generation model, shifting the retrieval objective from ranking documents by Search Satisfaction to constructing reliable context for correct answer generation. We formulate this shift as answer-oriented context construction through a three-stage framework: (1) Answer Support identifies candidate documents that contribute information to answer generation; (2) Content Trustworthiness assesses whether this information provides a reliable basis for correct answers from source, temporal, and factual perspectives; and (3) Context Organization selects, consolidates, and structures retained information under a finite context budget for consistent and robust generation. We further develop an industrial workflow spanning prior and posterior optimization and establish a systematic evaluation protocol covering both retrieval-side context and final answers. Experiments show consistent improvements at both Retrieval and Answer levels, demonstrating the effectiveness of the framework and its industrial implementation.

cs.IR↗