Skip to content

Watching background work, by kind

Scheduled jobs, queue consumers, background webhooks, long-running workers and CLIs: where to catch errors, when to report unknown, who retries.

Markdown
On this page

Background work fails quietly: nobody is looking at a screen when a queue stalls at 3 a.m. This guide shows, for each common kind of work, what to report, where to catch errors, and when a result is unknown rather than failed. It uses monitors for runs and ordinary error capture for exceptions.

Three rules hold everywhere:

  • Catch at the boundary. Report an error where the unit of work ends (one job, one message, one webhook), not deep inside helpers — otherwise the same failure is reported many times.

  • "Accepted" is not "done". HTTP 200 from your own endpoint or from a provider only means the request was received. Report success when the effect is confirmed.

  • Retries belong to your app. Belvador records attempts and the final result; it never re-runs anything.

Scheduled jobs (cron)#

One monitor per job, with the schedule it really runs on. Wrap the whole job in withMonitor (or the Sentry SDK's crons API, or curl):

  • Nothing to do this time → run.skip("nothing_due"), which is healthy. Do not report ok for a run that silently did nothing because a precondition failed — throw or report an error instead.

  • Partial success (500 of 520 invoices sent) → report the failures you know about as errors with captureException, and decide whether the run is ok or failed by your own rule. Write that rule down in the code.

Queue consumers#

A queue has no schedule; the risk is that consumption stops. Use an Every N minutes/hours monitor with a heartbeat from the consumer loop — for example once per batch or once per minute while idle — so a dead consumer shows up as Missed.

  • Do not create a run per message: thousands of runs per hour hide the signal and hit the 60-per-minute limit. If one specific message type matters, give it its own monitor.

  • For per-message retries, pass runId (the message id), attempt and willRetry so a message that succeeds on the third try is one healthy run.

  • A message moved to a dead-letter queue is a failure: capture it as an error with the message type as a tag (never its body).

Webhooks processed in the background#

Acknowledge fast, process later — and monitor the processing, not the acknowledgement:

  1. The HTTP handler stores the webhook and returns 200. Errors here are request errors, reported by the normal SDK.

  2. The background step that applies it runs inside withMonitor (on-demand monitor: When something happens), so failures and timeouts are visible even though nothing is ever "late".

When the background step calls a provider and the call times out after the request was sent, the effect is unknown: call run.unknown("provider_timeout"). The monitor shows Needs review until someone checks.

Long-running workers and agents#

A process that runs for hours or days cannot be one run. Report health with heartbeat every few minutes on an every-N monitor, and wrap each unit of work it performs (one sync, one export) in its own withMonitor if those units have their own expectations. Set the maximum runtime of a unit monitor to what a unit can really take; a hung unit then shows Timed out.

Short processes and CLIs#

A process that exits right after its work must send before it exits. With a Belvador or Sentry SDK, call await flush() (Python: sentry_sdk.flush()) at the end; with curl, the request itself is synchronous. Serverless functions with a hard time limit should report the start early so a function killed by its platform shows up as Timed out.

What not to do#

  • Do not send check-ins from browsers — they are refused, and a closed tab is not a failed job.

  • Do not put customer data in slugs, reasons or tags. Use ids and short codes.

  • Do not make a job fail because Belvador is unreachable. The SDKs and curl -fsS already never fail your work.

Choosing the signal#

You want to knowUse
A job threw or reported an errorError capture (automatic inside withMonitor)
A job did not run, ran too long, or its result is unknownA monitor
Which request in the browser led to a server errorTrace headers
Something unusual happened but nothing brokeA warning event — see Warnings and other signals

Posts on this topic

All posts

Spotted something wrong or missing? Tell us: hello@belvador.dev