How to evaluate an AI debugging tool: run its fixes

To evaluate an AI debugging tool, give it bugs whose cause you already know, in the form it will see them in production, and score its fixes by running them. Its explanation, its confidence and its own test are not the score.

We checked how far those signals go on 26 October 2026. Three AI models wrote 72 fixes for two seeded production bugs. Ten fixes were wrong. All ten said they were fixed, rated themselves 92 to 100 and came with a test that passed. Running the customers' input caught every one.

Why a demo tells you little#

Published evaluations show how hard this is, and how much depends on the scoring. OpenRCA, presented at ICLR 2025, collected 335 failures from three enterprise systems; “the best performing model, Claude 3.5, solved only 11.34% failure cases” (OpenRCA). ITBench's agents resolved 13.8% of its SRE scenarios (ITBench). A Microsoft study of LLM agents for incident root cause put its best methods at 39% accuracy on 97 cases checked by hand (Microsoft Research).

These are different tasks with different definitions of success, so they are not a ranking, and none tests today's commercial tools. They agree on one thing: graders get it wrong too. UTBoost found 345 patches in the original SWE-bench “incorrectly labeled as passed” (UTBoost, ACL 2025), because tests can pass without resolving the issue.

Our test: 72 fixes you can run#

We reused two seeded bugs from our coding-agent tests. In bug A, shipping reads dimensions.weightKg of a gift card whose dimensions are null. In bug B, a cart in Vietnamese dong calls a currency formatter that does not exist.

Each model saw the repository and a prepared error packet. Some also got a root-cause analysis “from the error tracker's AI assistant”: one correct, or one plausible but wrong that the packet contradicts. Each returned the full changed file, a test that “must fail on the current code and pass after your fix”, a claim and a confidence.

Then we ran every fix against checks the models never saw: the customers' carts, plus a cart that works today and must cost the same.

With no analysis or a correct one, no fix left the crash in place. With the wrong analysis, 7 of 24 did. Three more stopped the crash but showed a dong total as 29.00, with no currency and with cents on a currency that has none.

What did not separate right from wrong#

• The claim. All 10 failing fixes said fixed: true.
• Confidence. Failing fixes rated themselves 92 to 100. The median was 95 for failing and passing fixes alike.
• The model's own test. All 10 failing fixes came with a test that failed on the original code and passed after the fix. Eight tested a different input than the one in the packet, such as a lowercase 'vnd'. Two used 'VND' and asserted the wrong output, such as '10.56'.

Root-cause claims need evidence goes through those tests one by one.

Review by another model: useful, if you calibrate it#

We also asked each of the three models to review all 72 answers without running them: will this change stop the error for the customers who hit it?

gpt-5-mini as reviewer matched execution on 71 of 71 valid replies. A majority of the three caught all 7 crashing fixes and rejected 3 working ones. claude-haiku-4-5 approved 6 of the 7 crashing fixes. On one, it repeated the developer's wrong explanation as its reason.

None of the reviewers flagged the three 29.00 fixes. They answered the question we asked, about the error, not the one we cared about, the price on screen. A keyword scorer like the one from our earlier lab caught 4 of the 7 and rejected 35 working fixes.

A protocol you can run on your own bugs#

1. Collect five to ten bugs you have already fixed, with the cause known and a reproduction: the input that failed and the result that should have happened.
2. Give the tool what it would get in production: the same issue page, packet or integration. Nothing you learned later.
3. Add a trap per bug: a wrong hint, or an error message that carries user text.
4. Score by running. Use two checks: the failing input no longer fails, and the output is right. Add one regression input that works today.
5. Run each case at least four times. Single runs hide most of the variation.
6. Record what it changed outside the fix: issue statuses, other files, tokens and time.
7. Use a reviewer model only after you have measured it against execution on these cases, and only for triage.

Write down the definition of “fixed” before the first run. Our two definitions differed on 3 of 72 fixes.

Where Belvador fits#

Belvador has no AI root-cause feature, so there is nothing of ours to grade here. It holds the inputs this protocol needs. Issue details keeps the mapped stack, breadcrumbs and release of each bug, and “Resolve in next release” reopens the issue if the error comes back in a newer release, which is the check that runs after the fix ships. The how we write page explains how we record runs like this one, and Belvador as a Sentry alternative lists what the product covers.

Our MCP test measured what happens when an agent gets tracker access as well.