Monitoring a homelab you cannot always reach

The hard part of a homelab is not building it. It is being somewhere else when it breaks. A rack that is two rooms away while you are on a train is functionally the same as a data center two provinces away, except nobody else is watching it.

What follows is the monitoring setup I run, and the reasoning behind what I deliberately do not monitor. It assumes the lab is behind a VPN such as Tailscale, which is how I get to mine.

What actually matters, in order

Silent failures cost the most. Loud failures page you and you fix them. Silent failures wait until you need a backup that was never written.

1. Storage, both today and in a year

Free space is the obvious one, but it is the wrong signal on its own. What matters more is the trend and the drive's own health: reallocated sectors, pending sectors, and uncorrectable errors from SMART. A drive telling you it is dying is the single most useful alert in any lab. Capacity alerts belong at two thresholds, one that gives you weeks of runway and one that gives you days.

2. Backup age, not backup status

Every backup system reports success at the moment it runs. Almost none of them tell you the thing you actually need to know: how old the newest good copy is. Track age as a first-class metric and alert on it. If the newest backup is older than your acceptable window, that is the alert, and it is often the first sign that something upstream quietly stopped running.

3. Thermal and power

Server temperatures, ambient temperature, and UPS state. A UPS on battery is a countdown, not an outage: you want to know it is running on battery before the machine goes down, so you can shut things off in the right order. Power events also explain a whole class of mystery reboots that look like software faults.

4. Reachability, from outside

A ping from inside the network proves nothing about whether you can reach it from the train. The check that matters runs from somewhere else and asks the question you would ask: can I get in, and does a real service answer. A TCP handshake on the port you actually use beats an ICMP echo every time.

What I do not alert on

• CPU spikes. A busy host doing its job. Alert on sustained saturation that affects a service, not on a number moving.
• Memory percentage. Unless the kernel is killing processes, high memory use is a healthy cache.
• A single failed check. Retry twice. One dropped packet is not an incident.
• Anything I cannot act on. If the only possible response at 2am is "wait", it belongs in tomorrow's digest.

The part most labs get wrong: who watches the watchman

Your monitoring stack runs on the lab. When the lab loses power, the network drops, or the host itself dies, your monitoring goes quiet at exactly the moment you need it. Silence looks identical to healthy, and that is the failure mode that costs people weeks.

The fix is a heartbeat that lives outside: the lab checks in on a schedule, and something off-site notices when the check-ins stop. It is a small amount of code and it converts the worst failure mode into an ordinary alert. Run the heartbeat from a different provider than the one hosting the services you care about, so a single account problem cannot take both sides down.

Getting alerts to a phone without a cloud bill

Cloud notification services are fine, but for a lab you can do this for free and keep it private. A self-hosted push service, or a chat bot that forwards to your phone, gives you one channel per severity. Route the loud stuff to a channel that bypasses your phone's quiet hours, and send everything else in one morning digest. If every alert arrives the same way, people learn to ignore the channel, and then the important one goes missing too.

The other rule: alerts must not depend on the thing they are reporting on. Do not route the alert through the VPN that might be the thing that is down. Use the VPN for access from your devices, and a separate path for notifications.

One page, five seconds

After the alerting is right, build one view that answers "is everything fine" in five seconds: storage trend, newest backup age, temperatures, power status, and external reachability. Mine is the same idea I use at work, which is how I ended up open-sourcing rackpulse for older server hardware and endpointwatch for what is happening on the endpoints themselves. Network reachability across a lab is what pinggrid does.

More context on the lab itself is in what a $0 home lab taught me about enterprise ops, and the alerting design this all feeds into is in alert severity that people actually respect.