Co-Design Bench delivers a sobering assessment of coding agents operating outside simplified test conditions. Researchers from Alibaba and several Asian universities built the benchmark from real user requests, and the strongest of 10 evaluated model endpoints fully completed only 26.4 percent of its 273 tasks. GPT-6 Astra averaged 69.9 percent strict completion, which meant most individual requirements were satisfied, yet 73.6 percent of tasks still had at least one criterion unmet.
Co-Design Bench: A benchmark built from real coding sessions
The research team drew on 37,463 deduplicated queries from 26,314 real coding sessions rather than relying on synthetic edit instructions or predefined operations. Candidates received partial credit for signs of past difficulty, including requests that users rephrased two or more times, repeated follow-ups or explicit rollbacks. Experts then refined the selected requests and wrote hidden grading rubrics, producing 2,331 expert-authored acceptance criteria.
Each task includes a runnable application in its original state, a normalized request and, where available, attachments and session history. The starting application matters because it establishes what must change and what must remain intact. Acceptance is reference-free, meaning the system does not require gold code or a target screenshot, while still allowing several implementations to satisfy the same requirement.
The paper, posted to arXiv in October 2026, divides the work into 2 tracks. Multimodal Editing contains 100 tasks focused on localized visual, structural and content edits, with optional images, audio, video or documents. Repository Editing contains 173 tasks requiring coordinated changes across components, routes, state, data flow or content. Both tracks use the same interface, evidence standard and Pass, Partial or Fail verdicts.
Strict grading exposes unfinished work
The evaluation protocol first builds and renders each candidate application in an isolated environment. A primary evaluator grades every hidden criterion using source differences, rendered pages and Playwright traces. A separate language model verifier then challenges Pass and Partial verdicts, searching for stubs, hardcoded outputs, unreachable entry points, failures under varied inputs and mismatches with the request.
Evidence audits add another constraint. Visual claims require successful renders and screenshots, while interaction claims require recorded browser actions and resulting state. Missing evidence aborts evaluation before scores are aggregated. The authors note that the verifier can still make errors, and 2 frontend experts separately reviewed rubric scores for 200 task and model outputs.
Three measurements emerge from this process. Strict Completion counts only criteria marked Pass. Graded Completion awards half credit for Partial verdicts. Full-Pass Rate counts a task only when every criterion passes. The results show why the distinction is commercially important: a model can satisfy most requirements and still leave a project needing human cleanup before it is ready for use.
Complex revisions remain the weak point
GPT-6 Astra led the shared agentic harness, but its results differed sharply by track. It reached 0.754 strict completion on Multimodal Editing and fully passed 41.0 percent of those tasks. On Repository Editing, it posted 0.667 strict completion and fully passed 17.9 percent. That difference points to the extra coordination required when an edit spans multiple parts of a codebase rather than a single interface region.
The authors found a consistent effect from edit breadth. Repository tasks spanning 1 or 2 change types averaged 0.654 pooled strict completion, compared with 0.591 for tasks spanning 3 or 4 types, a gap of 6.3 points. Every 1 of the 10 endpoints performed worse on the broader group. These intertwined requirements are normal in real projects, the researchers said, which makes the result more consequential than an isolated benchmark concern.
Repository tasks are demanding by design. According to the paper, 166 of the 173 repository tasks require more than 1 type of change. The track contains 1,778 criteria, averaging 10.3 per task, while Multimodal Editing contains 553 criteria, averaging 5.5 per task. Structural, stylistic, interactive and content changes overlap across the repository set, so a partial result can reflect a failure in any part of a connected revision.
More tokens did not mean better results
Longer reasoning traces did not solve the problem. On Multimodal Editing, GLM-5.2 used about 7.29 million tokens per case against GPT-6 Astra’s 1.24 million, yet its strict completion was lower at 0.572. GLM-5.2 also needed a median of 69 turns, compared with 32 for GPT-6 Astra. The comparison suggests that additional interaction and token consumption cannot compensate for weak execution under compound requirements.
Other endpoints also trailed the leader. Claude Opus 5 met 66.1 percent of criteria on average and fully passed 18.3 percent of tasks. Qwen3.8-Max met 65.3 percent and fully passed 16.5 percent. Grok 4.6 finished last, meeting 51.1 percent and fully passing 6.6 percent. The spread demonstrates that strong average criterion coverage did not reliably convert into finished applications.
The ranking was not uniform across tracks. Qwen3.8-27B ranked second on repository edits at 0.645 strict completion, narrowly ahead of Claude Opus 5 at 0.642. The authors cautioned against deriving a general advantage from that result, because the harness comparison showed that execution environments can reshape model rankings. For buyers, the practical conclusion is to evaluate models and harnesses as a pair rather than treating a model name as a complete procurement answer.
Harness choice changes the leaderboard
The paper’s separate harness test provides the clearest warning against comparing isolated model scores. Qwen3.8-27B scored 0.749 on repository edits under opencode and 0.560 under qwen-code. The gap between Qwen3.8-27B and Kimi K3 shifted by up to 16.2 points depending on the harness. The authors therefore recommend reporting the model and working environment together.
Execution regime also matters independently of the harness. On the same 150 WebCompass-edit tasks, with the feature rubric, judge and render pipeline held fixed, qwen3.7-plus rose from 73.6 percent mean graded feature completion in a single-shot setting to 95.1 percent under agentic execution. Older edit benchmarks such as WebCompass-edit show agentic coding models scoring above 99 percent, with Co-Design Bench leaving a gap of more than 35 percentage points.
That contrast is the benchmark’s main argument. Earlier tests can become saturated when agents can run, check and repair their own code, leaving little room to distinguish models. Real user requests, by contrast, often combine interpretation, several coordinated changes and functional behavior, exposing a divide between partial requirement satisfaction and complete task success.
What the results mean for adopters
For companies deploying coding agents, the research offers a warning against treating high average criterion scores as delivery guarantees. A 69.9 percent average can coexist with a 26.4 percent full-pass rate, meaning project leads may still face substantial manual review. The benchmark also suggests that complexity, rather than agent effort alone, is a decisive factor: every tested endpoint weakened when tasks demanded 3 or 4 types of changes.
The findings have limits. Co-Design Bench evaluates 1 fixed request per task, so it does not test follow-up questions or mid-task corrections. The authors also acknowledge that AI judges can make mistakes. METR research cited by The Decoder found that the actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on the widely used SWE-bench Verified, illustrating how benchmark performance and real-world usability can diverge.
Even with those caveats, the central message remains clear. Co-Design Bench shows that today’s coding agents can handle substantial portions of real application revisions without delivering complete work. For adopters, that argues for evaluating the full system, including the harness, tracking full completion separately from average criterion coverage and budgeting for human cleanup on complex projects. Co-Design Bench turns near-perfect benchmark scores into a narrower claim, and that distinction matters for anyone deciding how much work an agent can safely be left to finish.