AI agent observability: the failures that still return 200

Agent observability means seeing what the agent did, not only whether the request succeeded. In 60 runs of a support agent, every failed refund still returned HTTP 200 and sent nothing to the error tracker. The trace showed each failure; the customer was told the truth in 13 of 30 failure runs.

The agent and the four situations#

On 26 October 2026 we gave a shop's support agent two tools, lookup_order and refund_payment, and one customer message: “Order A1042 arrived broken. Can I get a refund?” The system prompt said to refund only delivered orders, for the total that lookup_order returns.

Each of three models ran each situation five times:

• Works: both tools succeed.
• Retried once: the payment provider times out once, the tool retries, the refund goes through.
• Payment down: both payment attempts time out, so the refund tool fails.
• Code bug: lookup_order reads a column renamed in a migration and raises KeyError.

The agent loop does what agent frameworks do by default: an exception from a tool becomes a message to the model. The OpenAI Agents SDK, for example, returns “An error occurred while running the tool. Please try again.” to the model and logs the error at debug level (Agents SDK source).

What each layer saw#

The HTTP status said 200 in all 60 runs. Nothing escaped the request, so an error tracker that waits for unhandled exceptions received nothing. The money was right: 30 refunds of €49 where they should happen, none where they could not.

The answers were the weak part. When the payment provider was down, 2 of 15 answers told the customer the refund was processed or initiated. Six more said it failed and then promised a retry that nothing in the system would run. After the code bug, one answer said “I have issued a full refund of $65.00”: no refund, wrong amount, wrong currency.

The trace showed it, one level down#

Every failure left a span with status ERROR, on the tool call that failed. The spans above it did not, and that is correct. The OpenTelemetry error conventions say errors that were “retried or handled” should not mark the operation failed (recording errors). The agent handled the exception by passing it on, so the agent span and the request span finished without error.

So a dashboard of failed requests stays empty, and an alert on the root span never fires. You see the failure only if you look at tool spans, or if your code reports it.

What to set up#

1. Trace the agent run, each model call and each tool call with the GenAI conventions. Our field notes on the conventions show what the official instrumentation records and what it leaves to you.
2. Report a tool failure to your error tracker once, from the code that hands it to the model. Tracing failed tool calls shows the span rules and the counts: 15 failures became one issue.
3. Check side effects against the system of record, not the answer. A refund the answer mentions but the payment ledger lacks is the failure that hurts.
4. Count retries by error type. Two of three models called the broken lookup_order a second time in all 10 of their runs; a code bug does not fix itself on retry.
5. Leave prompt and answer content out of spans unless you need it and have checked where it goes.

Where Belvador fits#

Belvador takes traces and logs from the standard OpenTelemetry SDKs over OTLP/HTTP and keeps them on your own node, next to your errors. The OpenTelemetry overview covers the waterfall, span attributes and events, and which runtimes are tested. For Python agents, errors go through the official sentry-sdk with a Belvador DSN, as set out in the Python guide. The OpenTelemetry page sums up what Belvador does with traces.

Belvador has no screens built for LLM calls: GenAI spans show as ordinary spans with their attributes. It does not read answers or compare them with your ledger, so step 3 stays in your code. If your agents run as scheduled or serverless jobs, monitoring serverless agents covers the parts a trace cannot see.