# Incidents: related alerts as one piece of work with an owner

> Incident rules group monitor, uptime, SLO and error alerts; take it, add notes, wait out recovery and close on its own, reopen on relapse; one PagerDuty or incident.io page per incident.

An **incident** groups related alerts into one piece of work with an owner. When a nightly job fails and then times out, you get one incident — **INC-12 · Nightly billing** — with both alerts linked, one message to each destination, and one place to take it, write notes and close it. An incident closes on its own **only after everything linked has stayed healthy** for a while.

Open **Incidents** in a workspace. Viewers can read incidents; members can take, assign, note and resolve them; admins change the incident rules.

## How alerts become an incident

**Alerts → Incident rules** decides which alerts open an incident and how related alerts join it. Every workspace starts with one rule:

| Setting | Default |
|---|---|
| Kinds | Monitor, uptime, SLO and error-issue alerts |
| Severity | Medium and above |
| Group by | **Same source and environment** — the same monitor, uptime check, SLO or issue in the same environment |
| Window | 30 minutes |
| Recovery wait | 15 minutes |

Rules are tried in order; the first enabled rule that matches an alert wins. You can narrow a rule to projects and environments, or group by **project and environment** so that every alert of one app in production lands in the same incident.

An alert that joins an incident does not send its own message. Destinations hear about the **incident** instead: opened, reopened, severity raised, escalated and resolved. Without a matching rule (or with the rule switched off), alerts are sent one by one as before — see [Alerts](https://belvador.dev/en/docs/alerts).

A maintenance window that suppresses an alert also keeps it out of incidents ([maintenance windows](https://belvador.dev/en/docs/monitors/maintenance)).

## States

| State | Means |
|---|---|
| **Open** | At least one linked alert is firing and nobody has taken it |
| **Acknowledged** | A person took it (**Take it**, shortcut `A`); they become the owner if there was none |
| **Monitoring recovery** | Every linked alert recovered; Belvador waits out the recovery wait before closing |
| **Resolved** | Closed. **Verified** when it stayed healthy through the recovery wait; **Not verified** when someone closed it by hand |

- A new alert during **Monitoring recovery** puts the incident back to **Open** silently — no second "opened" message.
- A related alert **within the window** after an incident was resolved (and at most 24 hours later) **reopens** it, with a "reopened" message. Later than that, a new incident with a new number opens.
- The same alert arriving twice (a node retry, a duplicate delivery) never opens a second incident; two alerts arriving at the same moment still open one.
- **Resolve manually** asks for a reason; it is kept on the timeline and sent with the "resolved" message.

## Severity

An incident's severity is the highest severity among its alerts that are still firing; the **peak** is kept. When severity rises (for example a `critical` monitor joins), destinations get one "severity raised" message per level. Critical and high incidents always go out at once, even during quiet hours.

## Messages and on-call tools

| Event | Who hears it |
|---|---|
| Opened, reopened, severity raised | The destinations of the alert rule that matched |
| Escalated | The escalation destination of the [notification policy](https://belvador.dev/en/docs/alerts/policies) |
| Acknowledged | **PagerDuty** only, so the page stops ringing |
| Resolved | Everyone who was told it opened |

PagerDuty and incident.io destinations receive one alert per incident (trigger, acknowledge, resolve with the same deduplication key), so your on-call tool shows one page, not one per alert. On the incident page, **Delivery** shows each message: **Accepted** means the service took the message; only a person taking the incident counts as acknowledged.

## Timeline and notes

Every change is on the timeline: opened, alerts joining or recovering, severity changes, who took it, owner and title changes, fallbacks, escalation, notes and the resolution. Notes (up to 2,000 characters) are shown to your team as plain text. Names and titles come from your own apps and are always displayed as text, never as links or markup.

**Impact** shows the users affected when the node reports it; **Unknown** never means zero.

## API and MCP

The same actions are in the [REST API](https://belvador.dev/en/docs/reference/api) and the [MCP server](https://belvador.dev/en/docs/mcp):

| Call | Does |
|---|---|
| `GET /v1/incidents`, `GET /v1/incidents/{incident_id}` | List and read incidents (links, timeline, delivery states) — scope `incidents:read` |
| `POST /v1/incidents/{incident_id}/ack` | Take it — scope `incidents:write` |
| `POST /v1/incidents/{incident_id}/resolve` | Resolve with a reason |
| `POST /v1/incidents/{incident_id}/notes` | Add a note |
| `PATCH /v1/incidents/{incident_id}` | Rename or change the owner |
| `GET`/`POST /v1/incident-rules`, `DELETE /v1/incident-rules/{rule_id}` | Incident rules — scopes `alerts:read` / `alerts:write` (admin) |

A token acts with its owner's current role: a viewer's token can read but never take or resolve an incident.