How I cut incident response from hours to minutes

From 2023 to 2025 I ran IT, networking, and building management system (BMS) operations for a plastics manufacturer with multiple sites, leading a team of six. When I started, a real incident could take hours to surface, and longer to reach the right person. The fix wasn't buying a fancier monitoring platform. It was deciding what each alert was worth, and who it should wake up.

The problem: everything was monitored, nothing was actionable

The systems were all being watched. Network gear had one tool, servers another, the BMS a third, production equipment a fourth. Every one of them could generate alerts. The result was the classic failure mode: because everything pinged everyone, nothing mattered to anyone. A chilled water fault on the BMS and a dead disk on a file server competed for the same attention. Response time was measured in hours because the signal was buried in noise.

The build: collect, enrich, route, escalate

I rebuilt the alerting pipeline in four layers. None of the layers were exotic. The design was the hard part, not the tooling.

1. Collect

One intake path for every source: network devices, VMware hosts, Windows servers, BMS controllers, and the production endpoints that mattered. If it could emit a status, it got a collector. This closed the gaps where things silently stopped being watched.

2. Enrich

Every event was tagged with context: which site, which system class, which service it affected, and a severity. A storage warning on a non-critical server and a production line controller fault are not the same event, and the pipeline stopped treating them like they were.

3. Route

Each severity had one owner. Operators got operational alerts. I got the ones that needed a decision. Management got a daily digest, not a 2am page. The rule that changed everything: an alert with no named owner is a dashboard, not an alert.

4. Escalate

If a critical alert wasn't acknowledged inside a defined window, it escalated up the chain automatically. Acknowledged but not resolved? Same thing. Escalation made the system honest: silence no longer meant "fine".

The result

Incident response went from hours to minutes. Not because alerts were faster, but because the right person saw the right alert the first time, with the context to act. The same design carried us through the decommissioning of one site and the commissioning of another, which is where this kind of discipline pays for itself: during a move, things break constantly, and you need to know which breaks matter.

Lessons that transfer anywhere

• Alert fatigue is the real enemy. More dashboards don't fix it. Routing and severity do.
• Name an owner for every alert. If you can't say who should act, you haven't finished building the alert.
• Escalation is a feature. Acknowledged-but-ignored is a state your system should track.
• Document the runbook before the crisis. The move taught me that the second value of monitoring is knowing what's normal, which you can't reconstruct under pressure.
• Monitoring is a people problem. The best pipeline fails if the on-call culture doesn't trust it.

I later open-sourced part of this philosophy in endpointwatch, a live-response endpoint monitor that watches processes, connections, and persistence and explains what changed in plain English. Same idea: collect everything, decide what matters, tell the person who can act.