| title | ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval |
|---|---|
| authors | Aaryan Kapoor, Md Abdullah Al Hafiz Khan |
| date | 2026-09-01 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.01865 |
| affiliation | Kennesaw State University |
| upvotes | 6 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D. |
cite
@misc{kapoor2026execretrieval,
title={ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval},
author={Aaryan Kapoor and Md Abdullah Al Hafiz Khan},
year={2026},
eprint={2609.01865},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.01865}
}