Specific Labs published Real-SWE on September 13, and the sourcing is the whole point. The tasks are not public GitHub issues. They were lifted from private production codebases the firm licensed from operating companies, and every one of them was originally assigned to an engineer drawing a salary.
Ten tasks. Eight model-and-harness pairs. Eight rollouts per pair per task, 640 scored runs. The top line is Fable 5.1 on Claude Code at 38.8%.
The full board
Specific Labs scored each model inside its native CLI, on the argument that this is how enterprise engineers actually work.
| Model | Harness | Resolution rate |
|---|---|---|
| Fable 5.1 | Claude Code | 38.8% |
| GPT-6 Astra | Codex CLI | 33.8% |
| Gemini 3.8 Flash | Gemini CLI | 31.2% |
| GLM 5.3 | Claude Code | 28.8% |
| Grok 4.6 | Grok Build | 23.8% |
| Muse Spark 1.3 | Muse Code | 23.8% |
| Kimi K3 | Kimi Code | 18.8% |
| GPT-5.6 Sol | Codex CLI | 16.2% |
Resolution rate is pass@1, averaged over eight independent runs.
The ranking is the least interesting row of numbers here. Look at the task side instead. Six of the ten tasks land under 15% across all models. One of them, an analytics stream reducer, went 0 for 64: eight models, eight tries each, no passes. A tax-jurisdiction task scored 3.1%, carried entirely by one lucky run from Fable 5.1 and one from GLM 5.3.
What makes these hard is not the algorithm
Two medians tell you the shape of the work. The instruction runs 1,742 characters, in line with DeepSWE and Terminal-Bench. The reference solution edits 11 files. On FrontierCode and DeepSWE, that figure is 6.
The sandboxes say the rest. A single task can expose an AWS emulator, Docker, Kubernetes, GitHub, Linear, PostgreSQL, MySQL, MongoDB, Redis, plus Slack, Intercom, Google Drive and ClickUp. The published tax instruction asks the agent to get invoice billing right when every merchant on the platform settles tax differently: some hold their own rate, some price each invoice against the buyer’s destination through a third-party tax authority, some collect nothing, an exempt customer is charged nothing regardless of merchant configuration, settled sales get filed back to the authority under the invoice number so the returns reconcile, and invoices between European parties carry both VAT registrations.
None of that is a reasoning problem. It is the stuff a person knows after enough time at one company.
The failure taxonomy carries more signal than the score
Specific Labs classified all 468 failed runs using the DeepSWE taxonomy, and the models break in visibly different places.
Grok 4.6 missed a stated requirement in 41 of its 61 failures, 67.2%. Gemini 3.8 Flash produced an integration error in 27 of 55, 49.1% — the right idea, wired into the surrounding system incorrectly. GPT-5.6 Sol built on an unverified assumption in 29 of 67, 43.3%: it guessed how the system worked and never checked the workspace.
Fable 5.1’s 49 failures spread more evenly. Missed requirement 18 (36.7%), integration error 17 (34.7%), unverified assumption 12 (24.5%), and only 2 regressions (4.1%).
One more pair of numbers deserves its own line. Of rollouts finishing in under 10 minutes, 70 of 98 failed, 71.4%. Of rollouts running 10 minutes or longer, 398 of 542 failed, 73.4%. Letting the agent think longer bought nothing.
The unit economics, and the line item that is missing
Estimated cost per rollout runs from $2.50 on Gemini 3.8 Flash to $6.96 on Fable 5.1. The curve is not monotonic: Gemini Flash bought 31.2% for $2.50 while GLM 5.3 bought 28.8% for $5.12, twice the spend for a slightly worse result.
Divide cost per rollout by resolution rate and you get the expected spend per task actually resolved. Fable 5.1: $6.96 / 0.388, roughly $18. Gemini 3.8 Flash: $2.50 / 0.312, roughly $8. Those two figures are our arithmetic on the published numbers, not Specific Labs’ own.
Eighteen dollars a fix still undercuts an engineer-hour by a wide margin. The account is missing a line, though. The 61.2% that did not resolve did not return blank. They returned a patch that looks finished, and somebody has to read it closely enough to work out which requirement it dropped.
Which seat this actually raises
Specific Labs makes one claim worth quoting flat: 99% of tokens in real-world enterprises are hidden from the frontier models.
That explains the scores, and it points at where software engineering goes next. On June 26 we covered the layoffs landing on managers rather than engineers. On September 9 we covered Cognition moving engineers into the overseer seat. Real-SWE fills in what that seat is watching for: five failure modes — a dropped requirement, an unverified assumption, a bad integration, a regression, a change delivered to a file the running application never calls.
Not one of those is visible by reading the diff alone. Catching a dropped requirement means knowing how your company defines an exempt customer. Catching an integration error means knowing what calls the thing downstream. Catching a wrong-file change means knowing which copy production actually runs.
So on the enterprise side, value is sliding off the act of writing code and onto whoever holds the company-specific context. Juniors take the squeeze first, because the way that context used to get built was small change by small change, and small changes now have a second supplier. Two roles get lifted at the same time: the person who can specify a task precisely enough for an agent to get it right, and the person who can see what a patch left out before it merges.
What to watch
Real-SWE is ten tasks across three named codebases. Small, and the benchmark does not hide it. What makes it worth tracking is the method rather than the score: once anyone is licensing private production repositories to write tasks, the flattering numbers on public benchmarks acquire a control group.
Two things to watch. Whether 38.8% rises or falls as the task set expands past a handful of companies. And whether any enterprise writes agent-patch human review into a formal process document, which would be the first time this failure taxonomy turns into a job.
