Specific has unveiled Real-SWE, a benchmark aimed at measuring how AI models perform against private enterprise codebases, in contrast to public test suites like SWE-bench that currently dominate coding agent evaluation. The core premise is straightforward: benchmarks built from public GitHub repositories are increasingly prone to partial memorization by large language models during training, which skews results and inflates apparent problem-solving capability.
By relying on proprietary code that is not indexed or accessible to the crawlers used to build training corpora, Real-SWE aims to provide a more accurate measure of what coding agents can actually accomplish when facing genuine software engineering tasks, complete with the internal dependencies, quirks, and conventions specific to each organization. This approach reflects a broader wariness toward public benchmarks, which are seen as increasingly susceptible to data contamination and poorly representative of real production environments.
Specific's positioning echoes concerns raised by several research teams and companies deploying coding agents in practice, who have noted a gap between scores reported on public benchmarks and actual performance on large, historically layered, and sparsely documented codebases. Such a gap complicates adoption decisions for engineering teams trying to objectively evaluate tools like large language model-based coding assistants.
Methodological details about how the dataset was assembled, the number of tasks covered, or which models have already been tested remain limited at this stage. It remains to be seen whether this initiative will establish itself as an independent reference point, in a field where benchmark credibility and reproducibility continue to be central concerns for objectively comparing models.