Incidents: related alerts as one piece of work with an owner

An incident groups related alerts into one piece of work with an owner. When a nightly job fails and then times out, you get one incident — INC-12 · Nightly billing — with both alerts linked, one message to each destination, and one place to take it, write notes and close it. An incident closes on its own only after everything linked has stayed healthy for a while.

Open Incidents in a workspace. Viewers can read incidents; members can take, assign, note and resolve them; admins change the incident rules.

How alerts become an incident#

Alerts → Incident rules decides which alerts open an incident and how related alerts join it. Every workspace starts with one rule:

Rules are tried in order; the first enabled rule that matches an alert wins. You can narrow a rule to projects and environments, or group by project and environment so that every alert of one app in production lands in the same incident.

An alert that joins an incident does not send its own message. Destinations hear about the incident instead: opened, reopened, severity raised, escalated and resolved. Without a matching rule (or with the rule switched off), alerts are sent one by one as before — see Alerts.

A maintenance window that suppresses an alert also keeps it out of incidents (maintenance windows).

States#

• A new alert during Monitoring recovery puts the incident back to Open silently — no second "opened" message.

• A related alert within the window after an incident was resolved (and at most 24 hours later) reopens it, with a "reopened" message. Later than that, a new incident with a new number opens.

• The same alert arriving twice (a node retry, a duplicate delivery) never opens a second incident; two alerts arriving at the same moment still open one.

• Resolve manually asks for a reason; it is kept on the timeline and sent with the "resolved" message.

Severity#

An incident's severity is the highest severity among its alerts that are still firing; the peak is kept. When severity rises (for example a critical monitor joins), destinations get one "severity raised" message per level. Critical and high incidents always go out at once, even during quiet hours.

Messages and on-call tools#

PagerDuty and incident.io destinations receive one alert per incident (trigger, acknowledge, resolve with the same deduplication key), so your on-call tool shows one page, not one per alert. On the incident page, Delivery shows each message: Accepted means the service took the message; only a person taking the incident counts as acknowledged.

Timeline and notes#

Every change is on the timeline: opened, alerts joining or recovering, severity changes, who took it, owner and title changes, fallbacks, escalation, notes and the resolution. Notes (up to 2,000 characters) are shown to your team as plain text. Names and titles come from your own apps and are always displayed as text, never as links or markup.

Impact shows the users affected when the node reports it; Unknown never means zero.

API and MCP#

The same actions are in the REST API and the MCP server:

A token acts with its owner's current role: a viewer's token can read but never take or resolve an incident.