How to evaluate a coding agent
Coding agents are the easiest agents to grade, because a computer can run the result: does the code run, and do the tests pass? A solution must make the tests for the issue pass and keep every other test passing, and after that, check the diff, because an agent can pass tests by deleting assertions or special-casing the test input. For your own agent, build tasks from your own repository's history, and keep two suites: a hard capability suite that shows progress and a regression suite that should stay near 100%.
Why coding agents are easier to grade
Most agent outputs need a judgment call. Code can be executed. Run it, run the tests, read the result.
SWE-bench made this the standard way to grade coding agents: each task is a real GitHub issue from an open-source Python project, with the real fix and its tests.
Two groups of tests
A solution must pass two groups of tests:
- FAIL_TO_PASS: tests that failed before the fix and must pass after it. They prove the issue is fixed.
- PASS_TO_PASS: tests that passed before and must still pass. They prove nothing else broke.
Both matter. An agent that fixes the issue and breaks three other features has not solved the task.
Tests can be wrong
When OpenAI built SWE-bench Verified in August 2024, 93 Python developers screened tasks and kept 500, because many original tests were too specific or unrelated to the issue. Even cleaned-up tests age: in February 2026, OpenAI checked 138 hard Verified tasks and found 59.4% had real problems, and 35.5% had tests so narrow they rejected correct solutions.
So when an agent fails a task, read the test before blaming the agent.
Build tasks from your own repository
Public benchmarks tell you how agents do on other people's code. For your agent, use yours:
- Take issues your team already fixed.
- Check out the commit before the fix.
- Keep the fix's tests as the grader (FAIL_TO_PASS), and the rest of the suite as PASS_TO_PASS.
- Run the agent in a sandbox with no internet, so it can't look up the real fix.
- Apply its patch and run both groups of tests.
Check the diff after the tests
Tests passing isn't the end. An agent can pass tests by:
- Deleting or weakening assertions.
- Special-casing the exact test input.
- Changing test files instead of the code.
- Catching and silencing the error.
Flag any change to test files, and any change to files unrelated to the task, for review.
Static checks as a second signal
Use linters, type checks and complexity measures as a second signal of quality. NIST's structured testing guide says McCabe's original limit of 10 for cyclomatic complexity per function has strong support. Complexity isn't part of the major coding agent benchmarks, and it's a useful warning in your own repo when an agent's fixes keep adding branches.
Watch for duplication
GitClear's study of 211 million changed lines found that duplicated code blocks rose about 8 times during 2024, as AI coding assistants spread. Tests don't catch copy-paste. A duplication check in CI does.
Two suites
- Capability suite: hard tasks, starts with a low pass rate, shows whether the agent is getting better. It doesn't block anything.
- Regression suite: tasks the agent already solves, should stay near 100%, runs on every change to the agent's prompts, tools or model.
A real example
SWE-bench Verified launched in August 2024, with GPT-4o solving 33.2% of tasks. By early 2026, top scores had gone from 74.9% to 80.9% in six months, and OpenAI found both flawed tests and leaked answers. They stopped reporting it and moved to SWE-bench Pro.
From the floor
In DevFix, the self-healing coding agent we built in AI Engineer HQ, the loop was plan, code, validate the syntax, execute in a Docker sandbox, and check with test cases, and when tests failed, a reflector node read the errors and sent the code back to be fixed, repeating until everything passed. In the demo, the first versions of a simple string-reversal function were weak, and later versions passed all the generated test cases.
DevFix also had a safety guard with rule-based filters, so the generated code couldn't include calls like removing files or executing arbitrary code, because of real incidents where coding agents deleted user data after being told not to.
Related
Sources
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Jimenez and others (2023)
- Introducing SWE-bench Verified, OpenAI (August 2024)
- Why we no longer evaluate SWE-bench Verified, OpenAI (23 February 2026)
- NIST SP 500-235, Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric
- Anthropic, Demystifying evals for AI agents