Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Real-SWE benchmark tests coding agents on licensed private enterprise codebases; top model Fable 5.1 resolves only 38.8% of tasks.
Real-SWE is a new benchmark evaluating frontier AI coding agents on tasks drawn from private production codebases licensed from real companies, spanning billing, tax calculation, and cross-service migrations. Fable 5.1 with Claude Code leads at 38.8% resolution rate (pass@1 over eight runs), followed by GPT-6 Astra Codex CLI at 33.8% and Gemini 3.8 Flash Gemini CLI at 31.2%. Tasks use native harnesses and realistic tooling including Docker, Kubernetes, PostgreSQL, Redis, and Linear; median reference solutions edit 11 files versus 6 for DeepSWE and FrontierCode.
- Tasks come from licensed private production codebases of real companies
- Fable 5.1 tops leaderboard at 38.8% resolution; GPT-6 Astra at 33.8%
- Evaluates model-plus-harness combos like Claude Code and Codex CLI
- Median solution edits 11 files; instructions median 1,742 characters
- Environments include Docker, Kubernetes, PostgreSQL, Redis, and Slack
Full article2,477 words · extracted from withspecific.com · click to collapse
September 2026
Introducing Real-SWE
Benchmarking frontier AI models on private, real-world, enterprise codebases.
01Introduction
Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product.
- Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.
- Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.
- Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.
Can a coding agent actually do the work of a software engineer in the real world?
1
Fable 5.1
Claude Code
Resolution rate: 38.8%
2
GPT-6 Astra
Codex CLI
Resolution rate: 33.8%
3
Gemini 3.8 Flash
Gemini CLI
Resolution rate: 31.2%
4
GLM 5.3
Claude Code
Resolution rate: 28.8%
=5
Grok 4.6
Grok Build
Resolution rate: 23.8%
=5
Muse Spark 1.3
Muse Code
Resolution rate: 23.8%
7
Kimi K3
Kimi Code
Resolution rate: 18.8%
8
GPT-5.6 Sol
Codex CLI
Resolution rate: 16.2%
| # | Model | Harness | Resolution rate |
|---|---|---|---|
| 1 | Fable 5.1 | Claude Code | 38.8% |
| 2 | GPT-6 Astra | Codex CLI | 33.8% |
| 3 | Gemini 3.8 Flash | Gemini CLI | 31.2% |
| 4 | GLM 5.3 | Claude Code | 28.8% |
| =5 | Grok 4.6 | Grok Build | 23.8% |
| =5 | Muse Spark 1.3 | Muse Code | 23.8% |
| 7 | Kimi K3 | Kimi Code | 18.8% |
| 8 | GPT-5.6 Sol | Codex CLI | 16.2% |
Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models.
We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.
Real company tasks require company-specific context
Correct billing depends on business rules and external services
Fix invoice billing so each business charges the right tax and exempt customers aren't taxed.
View full instructionHide full instruction
Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL.
Services in the sandbox
- TaxJar sandbox
- TaxJar production
- InfluxDB ledger
- NestJS service
- TypeScript
Agents work across code, infrastructure, and business tools
Tools and services across Real-SWE task environments. Each task exposes only the services its workflow needs.
- AWS emulator
- Docker
- Kubernetes
- GitHub
- Linear MCP
- PostgreSQL
- MySQL
- MongoDB
- Gel
- Redis
- Go
- Python
- Node.js
- Vitest
- Slack
- Intercom
- Google Drive
- ClickUp
Codebase Selection
We selected codebases through a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads. The sample tasks analyzed below come from these codebases, including:
- A Luma/Partiful competitor with 200K+ users and a top 100 App Store ranking
- A consumer fintech platform processing 100K+ bank statements
- Enterprise AI sales platforms supporting complex business workflows
We prioritize code written to meet an actual user or business need over code written solely to create a benchmark task. Production engineering requires understanding existing architecture, preserving behavior that users rely on, and making changes within real operational constraints.
Brief instructions can require changes across many files
Our tasks describe the change needed, leaving agents to discover implementation details in the codebase and surrounding tools. Any behavior required by the verifier must be stated or reasonably discoverable. This leads to our prompts being slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions.
The work is cross-functional and complex: a single change can span multiple parts of the application. Agents must understand existing business logic and company coding patterns while keeping the surrounding system working.
Prompt length · median
A typical Real-SWE instruction is 1,742 characters.
FrontierCode2,056 chars
DeepSWE1,975 chars
Terminal-Bench 31,584 chars
FrontierSWE v2992 chars
Real-SWE1,742 chars
Files edited by the reference solution · median
11 files in Real-SWE, compared with 6 in FrontierCode and DeepSWE.
FrontierCode6
DeepSWE6
Real-SWE11
Models fail even in short rollouts.
71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts.
Triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns is difficult.
70/98 failed
398/542 failed
- Failed
- Passed
Every task is inspired or lifted verbatim from a private, real-world codebase. We find these types of tasks super interesting for three reasons:
- Tasks on private codebases are natively out of distribution. These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model. 99% of tokens in real-world enterprises are hidden away from the frontier models.
- These tasks are economically viable work. Each task here has a direct relationship to spend and was assigned to an engineer earning a salary. Most benchmarks test interesting, experimental capabilities that are often unlikely to be widespread in the real-world.
- Company-specific engineering patterns matter. Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. Many enterprises care about code standards and patterns. We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions.
02Analysis
Here's an analysis of a small sample of tasks from our benchmark. If you're interested in the sample, request access here.
6 of 10 tasks have resolution rates below 15%
| Task | Fable 5.1 | GPT-6 Astra | Gemini 3.8 Flash | GLM 5.3 | Grok 4.6 | Muse Spark 1.3 | Kimi K3 | GPT-5.6 Sol | Resolution rate |
|---|---|---|---|---|---|---|---|---|---|
| Multi-region sweep | 7 / 8 | 8 / 8 | 8 / 8 | 2 / 8 | 3 / 8 | 8 / 8 | 2 / 8 | 5 / 8 | 67.2% |
| API keys & environments | 8 / 8 | 5 / 8 | 7 / 8 | 5 / 8 | 4 / 8 | 6 / 8 | 0 / 8 | 7 / 8 | 65.6% |
| Entitlement overage lines | 8 / 8 | 7 / 8 | 5 / 8 | 3 / 8 | 1 / 8 | 1 / 8 | 6 / 8 | 1 / 8 | 50.0% |
| Customer identity migration | 3 / 8 | 1 / 8 | 3 / 8 | 4 / 8 | 8 / 8 | 3 / 8 | 4 / 8 | 0 / 8 | 40.6% |
| Billing schedule migration | 3 / 8 | 1 / 8 | 2 / 8 | 2 / 8 | 0 / 8 | 0 / 8 | 1 / 8 | 0 / 8 | 14.1% |
| API token metering | 1 / 8 | 5 / 8 | 0 / 8 | 1 / 8 | 0 / 8 | 0 / 8 | 1 / 8 | 0 / 8 | 12.5% |
| S3 datastore measurement | 0 / 8 | 0 / 8 | 0 / 8 | 3 / 8 | 2 / 8 | 1 / 8 | 1 / 8 | 0 / 8 | 10.9% |
| Linearizable scan | 0 / 8 | 0 / 8 | 0 / 8 | 2 / 8 | 1 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 4.7% |
| Tax jurisdiction | 1 / 8 | 0 / 8 | 0 / 8 | 1 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 3.1% |
| Analytics stream reducer | 0 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 0 / 8 | 0.0% |
Missed requirements are the most common failure
Failures are grouped by observed submission behavior using the same taxonomy across models, following DeepSWE.
Fable 5.1
GPT-6 Astra
Gemini 3.8 Flash
GLM 5.3
Grok 4.6
Muse Spark 1.3
Kimi K3
GPT-5.6 Sol
Unverified assumptionMissed requirementIntegration errorRegressionWrong file
No model solves every task
One square per rollout: each row is a task, each column a trial, eight trials per task for every model.
Fable 5.1
01
02
03
04
05
06
07
08
09
10
GPT-6 Astra
01
02
03
04
05
06
07
08
09
10
Gemini 3.8 Flash
01
02
03
04
05
06
07
08
09
10
GLM 5.3
01
02
03
04
05
06
07
08
09
10
Grok 4.6
01
02
03
04
05
06
07
08
09
10
Muse Spark 1.3
01
02
03
04
05
06
07
08
09
10
Kimi K3
01
02
03
04
05
06
07
08
09
10
GPT-5.6 Sol
01
02
03
04
05
06
07
08
09
10
PassUnverified assumptionMissed requirementIntegration errorRegressionWrong file
Different models fail in different ways
Percentages are out of each model's failed runs, not all runs.
03Effort & the Frontier
Estimated frontier
Resolution rate (%)1015202530354045$2$3$5$10Cost per rollout (USD, log scale)Gemini 3.8 Flash: 31.2% · $2.50; Gemini CLIGemini 3.8 Flash31.2% · $2.50GPT-5.6 Sol: 16.2% · $2.65; Codex CLIGPT-5.6 Sol16.2% · $2.65Muse Spark 1.3: 23.8% · $2.74; Muse CodeMuse Spark 1.323.8% · $2.74Grok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higherGrok 4.623.8% · $3.44Kimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higherKimi K318.8% · $3.90GPT-6 Astra: 33.8% · $4.67; Codex CLIGPT-6 Astra33.8% · $4.67GLM 5.3: 28.8% · $5.12; Claude CodeGLM 5.328.8% · $5.12Fable 5.1: 38.8% · $6.96; Claude CodeFable 5.138.8% · $6.96
Estimated rollout costs range from $2.50 to $6.96
| Rank | Model | Estimated cost (USD) |
|---|---|---|
| 1 | Gemini 3.8 Flash | $2.50 |
| 2 | GPT-5.6 Sol | $2.65 |
| 3 | Muse Spark 1.3 | $2.74 |
| 4 | Grok 4.6 | $3.44 |
| 5 | Kimi K3 | $3.90 |
| 6 | GPT-6 Astra | $4.67 |
| 7 | GLM 5.3 | $5.12 |
| 8 | Fable 5.1 | $6.96 |
mean per rollout, by task
Swipe the chart to see all tasks.
0100k200k300k400k01020304050607080910taskEntitlement overage lines · Fable 5.1: 34kMulti-region sweep · Fable 5.1: 30kTax jurisdiction · Fable 5.1: 78kAPI token metering · Fable 5.1: 95kAPI keys & environments · Fable 5.1: 71kS3 datastore measurement · Fable 5.1: 62kCustomer identity migration · Fable 5.1: 67kBilling schedule migration · Fable 5.1: 26kLinearizable scan · Fable 5.1: 86kAnalytics stream reducer · Fable 5.1: 88kEntitlement overage lines · GPT-6 Astra: 13kMulti-region sweep · GPT-6 Astra: 13kTax jurisdiction · GPT-6 Astra: 24kAPI token metering · GPT-6 Astra: 31kAPI keys & environments · GPT-6 Astra: 32kS3 datastore measurement · GPT-6 Astra: 22kCustomer identity migration · GPT-6 Astra: 25kBilling schedule migration · GPT-6 Astra: 15kLinearizable scan · GPT-6 Astra: 33kAnalytics stream reducer · GPT-6 Astra: 29kEntitlement overage lines · Gemini 3.8 Flash: 78kMulti-region sweep · Gemini 3.8 Flash: 67kTax jurisdiction · Gemini 3.8 Flash: 95kAPI token metering · Gemini 3.8 Flash: 134kAPI keys & environments · Gemini 3.8 Flash: 102kS3 datastore measurement · Gemini 3.8 Flash: 106kCustomer identity migration · Gemini 3.8 Flash: 97kBilling schedule migration · Gemini 3.8 Flash: 70kLinearizable scan · Gemini 3.8 Flash: 106kAnalytics stream reducer · Gemini 3.8 Flash: 88kEntitlement overage lines · GLM 5.3: 68kMulti-region sweep · GLM 5.3: 53kTax jurisdiction · GLM 5.3: 141kAPI token metering · GLM 5.3: 177kAPI keys & environments · GLM 5.3: 125kS3 datastore measurement · GLM 5.3: 121kCustomer identity migration · GLM 5.3: 90kBilling schedule migration · GLM 5.3: 58kLinearizable scan · GLM 5.3: 172kAnalytics stream reducer · GLM 5.3: 169kEntitlement overage lines · Grok 4.6: 7kMulti-region sweep · Grok 4.6: 3kTax jurisdiction · Grok 4.6: 12kAPI token metering · Grok 4.6: 15kAPI keys & environments · Grok 4.6: 16kS3 datastore measurement · Grok 4.6: 13kCustomer identity migration · Grok 4.6: 20kBilling schedule migration · Grok 4.6: 6kLinearizable scan · Grok 4.6: 261kAnalytics stream reducer · Grok 4.6: 315kEntitlement overage lines · Muse Spark 1.3: 36kMulti-region sweep · Muse Spark 1.3: 43kTax jurisdiction · Muse Spark 1.3: 67kAPI token metering · Muse Spark 1.3: 152kAPI keys & environments · Muse Spark 1.3: 104kS3 datastore measurement · Muse Spark 1.3: 71kCustomer identity migration · Muse Spark 1.3: 76kBilling schedule migration · Muse Spark 1.3: 38kLinearizable scan · Muse Spark 1.3: 141kAnalytics stream reducer · Muse Spark 1.3: 137kEntitlement overage lines · Kimi K3: 30kMulti-region sweep · Kimi K3: 9kTax jurisdiction · Kimi K3: 39kAPI token metering · Kimi K3: 69kAPI keys & environments · Kimi K3: 44kS3 datastore measurement · Kimi K3: 32kCustomer identity migration · Kimi K3: 66kBilling schedule migration · Kimi K3: 19kLinearizable scan · Kimi K3: 71kAnalytics stream reducer · Kimi K3: 55kEntitlement overage lines · GPT-5.6 Sol: 12kMulti-region sweep · GPT-5.6 Sol: 8kTax jurisdiction · GPT-5.6 Sol: 22kAPI token metering · GPT-5.6 Sol: 31kAPI keys & environments · GPT-5.6 Sol: 25kS3 datastore measurement · GPT-5.6 Sol: 25kCustomer identity migration · GPT-5.6 Sol: 24kBilling schedule migration · GPT-5.6 Sol: 13kLinearizable scan · GPT-5.6 Sol: 37kAnalytics stream reducer · GPT-5.6 Sol: 30k
Fable 5.1 · 64k overallGPT-6 Astra · 24k overallGemini 3.8 Flash · 94k overallGLM 5.3 · 117k overallGrok 4.6 · 67k overallMuse Spark 1.3 · 87k overallKimi K3 · 43k overallGPT-5.6 Sol · 23k overall
View task values
Task- Fable 5.134k
- GPT-6 Astra13k
- Gemini 3.8 Flash78k
- GLM 5.368k
- Grok 4.67k
- Muse Spark 1.336k
- Kimi K330k
- GPT-5.6 Sol12k
04Evaluation Setup
Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.
Text extracted automatically; images, tables and formatting may be missing. Original: https://withspecific.com/benchmarks/real-swe