Blog
Alert severity that people actually respect
Every alerting system I have inherited had the same problem: too many severity levels, and no agreement on what they meant. One tool called a disk warning critical. Another called a production line fault a notice. The person on call learned to distrust all of it, which is the only rational response to a system that cries wolf.
Severity is not a label you attach to an event. It is a decision about who gets interrupted, and for how long you are willing to let something sit. Once I started treating it that way, the number of levels collapsed to four and the arguments stopped.
I wrote about the wider pipeline in how I cut incident response from hours to minutes. This post is the part people ask about most: the severity model itself.
Start with the interruption, not the event
The question is never "how bad is this event". It is "who needs to stop what they are doing, right now, because of this". That reframes everything. A dead disk in a replicated array is not the same event as a dead disk on a single point of failure, even though the monitoring tool reports both the same way.
The four tiers
Four levels is enough for a mid-sized operation. More than that and people stop remembering which is which.
| Tier | What it means | Who gets it | How | Ack window |
|---|---|---|---|---|
| P1 | Something is down or about to cause damage. Operations or safety affected now. | Named owner, then their backup | Page | 15 minutes |
| P2 | Degraded, but the business is still running. Will become P1 if ignored. | Named owner | Push or DM | Next business morning |
| P3 | Needs a human decision eventually. Capacity, wear, licence, documentation drift. | Queue owner | Ticket or daily digest | Days |
| P4 | Record it and move on. Nobody is interrupted, ever. | Nobody | Log and dashboard | None |
The P4 tier is the one people forget to create, and it is the most important release valve. Without a place for noise to go, noise lands in P1 and P2 and ruins them.
Five rules that make the model hold
1. One owner per severity, by name
Not a team, not a shared inbox, a person. A shared inbox means nobody owns it. If you cannot name who should act on an alert, the alert is not finished being designed.
2. Severity comes from impact, not from the source system
Normalize on the way in. A network switch alerting "critical" because a port flapped does not get to define critical. Your severity assignment happens at the pipeline layer, where you know what the device does for the business.
3. Every tier has a different channel
P1 pages. P2 pushes. P3 becomes a ticket. P4 is a dashboard nobody watches until they need it. If two tiers share a channel, they are one tier pretending to be two.
4. Acknowledged is not resolved
Track the two states separately. An alert acknowledged and then forgotten is worse than one that never got a response, because the system thinks it is handled. Wire a second timer to ack-without-resolution.
5. Escalation is automatic and boring
No ack in the window, the next person gets it. No drama, no manual chase. The escalation path is documentation, and it should work at 3am without you.
Anti-patterns I have had to undo
- Severity inflation. Someone marks everything high because they want it noticed. Six months later every alert is high and none of them are.
- Alerting the whole team. Broadcasting is not ownership. It is a way of guaranteeing that the person who could fix it assumes someone else has it.
- Dashboard-as-alert. If it requires someone to be looking at a screen, it is a report, not an alert.
- Thresholds set from the vendor's default. Your disk, your temperature, your ack window. Defaults are a starting point for a conversation, not a policy.
- No quiet hours for the non-critical tiers. A P3 at 2am trains people to mute the channel that P1 needs.
Rolling it out in a week
You do not need a migration project for this. In practice: inventory every alert source and write down who would act on each one today. Anything with no answer goes to P4 until someone claims it. Then assign tiers by impact, set the ack windows, and run in shadow mode for a week so people can see what would have paged them. Only then turn paging on.
The measure of success is not fewer alerts. It is that when the phone buzzes, the person who picks it up believes it matters. That belief is the whole system.
If you want the operational end of this, the tooling I built for it is open source: endpointwatch watches endpoints and explains what changed in plain English, and rackpulse does the same for older server hardware.