Watching background work, by kind
Scheduled jobs, queue consumers, background webhooks, long-running workers and CLIs: where to catch errors, when to report unknown, who retries.
On this page
Background work fails quietly: nobody is looking at a screen when a queue stalls at 3 a.m. This guide shows, for each common kind of work, what to report, where to catch errors, and when a result is unknown rather than failed. It uses monitors for runs and ordinary error capture for exceptions.
Three rules hold everywhere:
Catch at the boundary. Report an error where the unit of work ends (one job, one message, one webhook), not deep inside helpers — otherwise the same failure is reported many times.
"Accepted" is not "done". HTTP 200 from your own endpoint or from a provider only means the request was received. Report success when the effect is confirmed.
Retries belong to your app. Belvador records attempts and the final result; it never re-runs anything.
Scheduled jobs (cron)#
One monitor per job, with the schedule it really runs on. Wrap the whole job in withMonitor (or the Sentry SDK's crons API, or curl):
Nothing to do this time →
run.skip("nothing_due"), which is healthy. Do not reportokfor a run that silently did nothing because a precondition failed — throw or report an error instead.Partial success (500 of 520 invoices sent) → report the failures you know about as errors with
captureException, and decide whether the run isokor failed by your own rule. Write that rule down in the code.
Queue consumers#
A queue has no schedule; the risk is that consumption stops. Use an Every N minutes/hours monitor with a heartbeat from the consumer loop — for example once per batch or once per minute while idle — so a dead consumer shows up as Missed.
Do not create a run per message: thousands of runs per hour hide the signal and hit the 60-per-minute limit. If one specific message type matters, give it its own monitor.
For per-message retries, pass
runId(the message id),attemptandwillRetryso a message that succeeds on the third try is one healthy run.A message moved to a dead-letter queue is a failure: capture it as an error with the message type as a tag (never its body).
Webhooks processed in the background#
Acknowledge fast, process later — and monitor the processing, not the acknowledgement:
The HTTP handler stores the webhook and returns 200. Errors here are request errors, reported by the normal SDK.
The background step that applies it runs inside
withMonitor(on-demand monitor: When something happens), so failures and timeouts are visible even though nothing is ever "late".
When the background step calls a provider and the call times out after the request was sent, the effect is unknown: call run.unknown("provider_timeout"). The monitor shows Needs review until someone checks.
Long-running workers and agents#
A process that runs for hours or days cannot be one run. Report health with heartbeat every few minutes on an every-N monitor, and wrap each unit of work it performs (one sync, one export) in its own withMonitor if those units have their own expectations. Set the maximum runtime of a unit monitor to what a unit can really take; a hung unit then shows Timed out.
Short processes and CLIs#
A process that exits right after its work must send before it exits. With a Belvador or Sentry SDK, call await flush() (Python: sentry_sdk.flush()) at the end; with curl, the request itself is synchronous. Serverless functions with a hard time limit should report the start early so a function killed by its platform shows up as Timed out.
What not to do#
Do not send check-ins from browsers — they are refused, and a closed tab is not a failed job.
Do not put customer data in slugs, reasons or tags. Use ids and short codes.
Do not make a job fail because Belvador is unreachable. The SDKs and
curl -fsSalready never fail your work.
Choosing the signal#
| You want to know | Use |
|---|---|
| A job threw or reported an error | Error capture (automatic inside withMonitor) |
| A job did not run, ran too long, or its result is unknown | A monitor |
| Which request in the browser led to a server error | Trace headers |
| Something unusual happened but nothing broke | A warning event — see Warnings and other signals |
Posts on this topic
All posts- Monitoring AI agents that run on serverless functionsAn AI agent on serverless is a chain of short functions with hard time limits. What to monitor at each step, with three lab runs and documented limits.
- Cron jobs that never finish: what your heartbeat actually seesWe stopped a cron job with SIGTERM and SIGKILL, three runs each. Where you send the heartbeat decides whether you get a failure, a false OK or silence.
- Cloudflare Workers error tracking: the errors that still return 200A test Worker failing five ways, three runs each: what the client, Cloudflare's Tail and an error-tracking wrapper saw, and how to catch waitUntil errors.
Spotted something wrong or missing? Tell us: hello@belvador.dev