Signal monitors: counts, assertions and stalled runs

Catch silent failures from the errors, check-ins, spans and logs you already send: too few or too many, wrong results, runs that stall — judged on your node.

Markdown

On this page

Choose what to watch (count and assertion)
When it alerts
Progress (stalled runs)
Metric monitors
Severity and alerts
From code, the API or an AI agent

Scheduled-job monitors catch work that does not run. Some failures are quieter: the job runs and reports "success", but nothing was published, a model returned empty answers, or a worker started and then froze. Signal monitors watch for those, using the errors, check-ins, spans, logs and metrics your apps already send.

Every signal monitor is judged on your node every 15 seconds, on the data the node stores. Belvador only receives the verdict (counts and thresholds, never messages or attribute values).

Type	Use it when	Example
Count	something must happen often enough — or not too often	"at least 1 log `post.published` every 60 min", "at most 50 error events on `queue=emails` every 5 min"
Assertion	results must be correct, not only "successful"	"spans `gen_ai.chat` must have `gen_ai.usage.output_tokens > 0`; alert at 3 failures in 10 min"
Deadline	scheduled work must finish before its due time	"each scheduled post must publish by its time + 5 min" — see Deadline monitors
Progress	running work must not stall	"an import that started must send a sign of life at least every 10 min"
Metric	a number must stay within limits	"p95 of `http.server.request.duration` over 5 min below 0.8 s", "`queue.oldest_age` below 600 s per queue"

Create one in Monitors → Create monitor, then pick the type.

Choose what to watch (count and assertion)#

Pick a signal — error events, check-ins, spans or logs — then up to 5 match conditions. Every condition must hold (AND). Names are the standard ones, so the SDK or OpenTelemetry setup you already have works unchanged:

Sentry fields: level, release, transaction, environment, exception.type, monitor.slug, tags.<name>
OpenTelemetry attributes: service.name, http.response.status_code, http.route, error.type, gen_ai.usage.output_tokens, the span name, the log event.name, severity_text
Belvador run fields: belvador.run.id, belvador.outcome

Operators: = ≠ < ≤ > ≥, is one of (up to 20 values), exists, is missing, starts with. Values are up to 256 characters. There are no regular expressions: the cost of checking one signal stays fixed whatever your data looks like.

An assertion adds up to 5 must be true conditions. A matching signal that fails any of them is a violation.

When it alerts#

Window: 1 to 1440 minutes. A window ends at every minute, so a problem is seen within a minute of the window closing.
Count alerts when the count is below a threshold, above it, or outside a range.
Assertion alerts when violations in the window reach the threshold.
When nothing matches in a whole window: count it as 0 (right for "at least N" — silence is the problem), alert, keep the last state, or treat as healthy.
Healthy again after: how many green windows in a row close the alert.

A node that is offline or still catching up is never judged: the monitor shows "can't tell" instead of a false alarm, and missing data is not read as zero.

Preview the last 30 days#

While you edit a count or assertion monitor, the right-hand panel replays the same rule on the last 30 days of data your node holds and shows how many times it would have alerted. Use it to pick a threshold before the first alert. The preview is read from your node with a short-lived ticket; nothing is copied to Belvador.

Progress (stalled runs)#

A run starts with a check-in in_progress, or with a span or log carrying belvador.run.id. Any later signal with the same id is a sign of life. When a running run sends nothing for longer than stalled after (1 to 1440 minutes), Belvador alerts MONITOR_STALLED. A new signal for that run closes the alert.

Metric monitors#

A metric monitor reads one metric your services send and judges one number per window:

Metric and aggregation — the same choices as the Metrics page (avg, sum, min, max, count, rate, p50–p99), with up to 5 attribute filters. Create monitor on the Metrics page carries the current query over.
Rule — the value is above a threshold, below it, or outside a range, in the metric's unit.
Group by — up to 2 attribute keys. Each value set (for example each http.route) is judged and alerts on its own, up to 50 groups per environment. Alerts name the monitor, never the group's values.
When a window has no points — keep the last state (default: a quiet exporter is not a recovery), count it as 0, alert, or treat as healthy.

Points arrive about once a minute, so a window is judged 120 seconds after it ends, once its last minute is complete. While you edit, the right-hand panel draws the last 24 hours of the same query, one point per window, with your thresholds and how many windows would have alerted. Metric monitors need agent 0.1.37 or later on your node.

Severity and alerts#

Each monitor has a severity (critical, high, medium, low) that travels with every alert. Who is told is set in Alerts → rules under Monitor incidents; the kinds are count out of range, check failed, deadline missed, stalled and metric out of range. Alerts carry the monitor, project, environment, the count and the threshold — never a message, an attribute value or a run id.

To silence alerts during a planned deploy, add a maintenance window.

From code, the API or an AI agent#

The same settings are accepted by POST /v1/monitors and PATCH /v1/monitors/<monitor_id> with a settings object (settings.type is count, assertion, deadline, progress or metric), and by the MCP tool that configures monitors. GET /v1/monitors/<id>/signal returns the minute counts, and POST /v1/monitors/backtest runs the same 30-day preview. See the API reference.

Spotted something wrong or missing? Tell us: hello@belvador.dev