Node health and metrics
What Healthy, Degraded and Unreachable mean, and how to read CPU, memory, disk and queue metrics.
On this page
The Node page shows whether your node is receiving and storing errors, and why not when it isn't. The node reports a heartbeat every 30 seconds; the page refreshes on its own, so values can be up to about a minute old.
Statuses#
| Status | Meaning | What to do |
|---|---|---|
| Healthy | Heartbeats arrive and every check passes. | Nothing. |
| Degraded | The node runs, but a check fails — the page lists which. | See the reasons below. Errors are still accepted. |
| Unreachable | No heartbeat for more than 2 minutes. | Errors are held at the edge meanwhile. See below. |
| Setting up | The node is being created. | Follow the timeline. |
| Upgrading | Signed node software is being installed. | Wait; it finishes on its own. |
| Failed | Setup or a lifecycle operation failed. | Open the timeline and choose Retry from this step. |
| Deleting | Deletion is scheduled or running. | Undo is possible during the first 15 minutes. |
Reasons for Degraded#
| Reason | Usual cause | Fix |
|---|---|---|
| Disk above 85% | More events than the machine's disk holds for your retention | Change machine size to a bigger one |
| Memory above 90% for 10 minutes | A sustained burst of events | Resize, or lower intake limits |
| ClickHouse write errors | The node's database cannot write | Usually recovers; if it stays, Replace machine & restore |
| Release manifest older than 24 hours | The node cannot reach Belvador's update channel | Check that nothing blocks outgoing traffic in the Hetzner project |
| More than 1,000 messages waiting in the outbox | The node cannot hand results to the edge fast enough | Usually clears on its own as the connection recovers |
Metrics#
| Field | What it shows |
|---|---|
| CPU, Memory, Disk | Current use of the machine |
| Ingest | Events per second being stored |
| Last heartbeat | When the node last reported in |
| Held at the edge | Errors the edge is keeping for the node — normally zero |
| Outbox | Results waiting to go from the node to the edge |
| Machine, Versions, Channel | Server type and location, node software versions and update channel |
No data yet means the node has not reported that value; it is not a status of its own.
The Security section shows how the Hetzner token is held, when it was last used, that no inbound ports are open, whether other servers exist in the same Hetzner project, and when the latest signed release manifest arrived. Activity lists what Belvador, members and the node itself did.
When the node is unreachable#
Your apps can keep sending. Belvador's edge accepts their errors and keeps them sealed with your node's own key — Belvador cannot read them — for up to 14 days and 8 GiB per node. When the node reconnects, the edge delivers them in order.
If the buffer fills up or data is older than 14 days, new errors are refused with 429 and lost errors are counted as Lost (buffer expired) in the project's intake numbers.
What to check:
In Hetzner Console, is the server running? Was it stopped, rebooted or deleted by someone?
Is the Hetzner account in good standing?
Wait a few minutes after a Hetzner incident — the node reconnects on its own.
If the machine is lost, use Replace machine & restore — see Updates and rebuilds.
While unreachable, issues cannot be viewed (the data lives on the node) and Fleet shows Unknown for projects whose numbers are older than 10 minutes.
Spotted something wrong or missing? Tell us: hello@belvador.dev