Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?
Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?
Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's…
- Tags
- ai-research
From the original article
1 min read at huggingface.co
Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning.
By Silin Chen, Yufei Yang, Xiaodong Gu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.