AI root-cause claims need evidence you can run

Believe an AI's root-cause claim when a test with the input production actually sent fails on the old code for the same reason, passes on the fix, and returns the result you expect. Add one input that works today. A test that merely fails and then passes is not enough.

We learned that from ten wrong fixes on 26 October 2026. Three AI models wrote 72 fixes for two seeded bugs, each with its own test. All ten wrong fixes came with a test that failed before the fix and passed after. The tests were real; they tested the wrong thing.

One wrong fix, with a passing test#

The bug: a cart in Vietnamese dong crashes checkout because there is no VND currency formatter. The model also got an AI analysis claiming the client sends 'vnd' in lowercase. It isn't: the breadcrumb in the same prompt read currency VND.

The fix uppercases the code before the lookup. The test checks 'usd' and 'eur', which fail on the old code with the same TypeError customers hit, and pass on the fix. 'VND' is already uppercase, so the customers' cart still crashes. Confidence: 99.

What the ten tests checked#

Two patterns covered all ten:

• A different input. Eight tests built the case the model believed in: a SKU missing from the catalog instead of the gift card whose dimensions is null, or lowercase 'vnd'. Same kind of error, different data.
• The wrong expectation. Two tests used 'VND' and asserted '10.56' and '100000.00': no crash, but a dong price with cents and no currency.

Tests can also fail the other way. Seven of 72 model-written tests failed against the model's own working fix. A test is a claim too.

This is not new to AI. UTBoost found 345 patches in the original SWE-bench “incorrectly labeled as passed”, because generated patches could “pass the tests without resolving the underlying issue” (UTBoost). A 2025 study of generated patches puts it plainly: a patch “may pass the tests but nevertheless fail to match the developers' expectations” (arXiv 2503.15223).

The evidence to require#

1. The failing input from production: the cart, request or job payload from the event, scrubbed. Not one the model made up.
2. The expected result, written by a person who knows the product. The model guessed '10.56' for a dong price.
3. The same failure on the old code: same error and same input, not just the same message.
4. A pass on the fix, and one input that works today and must keep working.
5. After release: the error does not come back. That is the check no test in a pull request can make.

For a coding agent, ask for it in the task: “Write the test from the input in this event, and tell me what the test asserts in plain words before you change any code.”

Where Belvador fits#

Belvador does not check fixes and has no AI root-cause feature. It keeps the two pieces this needs. The event on issue details holds the breadcrumbs, tags and request context, scrubbed, that a reproduction starts from. “Resolve in next release” reopens the issue if the error returns in a newer release, the check after the pull request. The Sentry alternative page covers the rest.

For the bigger picture, see how to evaluate an AI debugging tool. Seer or a coding agent shows where wrong analyses come in, and Claude Code and production errors shows what to hand an agent so it starts from the right input.