I maintain three servers, a NAS, and a handful of Docker containers that run things I actually care about. For two years I checked them manually like an idiot. Open Grafana, look at graphs, close Grafana, repeat the next day.
Then I wrote a Python script that does it for me and sends a Telegram message with anything weird. Here it is.
What It Checks
Four things, nothing more:
1. Disk usage on every mount point ( warns at 80%, alerts at 90% )
2. Docker container status ( anything not "running" or "healthy" )
3. Memory pressure ( available RAM below 500MB )
4. CPU load average over 15 minutes ( above 4.0 on a 4-core box )
That is it. I do not need Prometheus alerting me that a pod restarted 6 hours ago. I need to know if something is broken RIGHT NOW.
The Script
Here is the full thing. I run it from cron every 15 minutes.
#!/usr/bin/env python3
"""Server health check — sends Telegram alerts."""
import subprocess, json, os, requests
from pathlib import Path
TELEGRAM_TOKEN = os.environ["TELEGRAM_BOT_TOKEN"]
CHAT_ID = os.environ["TELEGRAM_CHAT_ID"]
THRESHOLDS = {
"disk_warn": 80,
"disk_alert": 90,
"mem_min_mb": 500,
"load_max": 4.0,
}
def check_disk():
"""Check all mount points with df."""
issues = []
result = subprocess.run(
["df", "-h", "--output=pcent,target"],
capture_output=True, text=True
)
for line in result.stdout.strip().split("\n")[1:]:
pct_str, mount = line.strip().split(None, 1)
pct = int(pct_str.strip("%"))
if pct >= THRESHOLDS["disk_alert"]:
issues.append(f"[ALERT] Disk {mount} at {pct}%")
elif pct >= THRESHOLDS["disk_warn"]:
issues.append(f"[WARN] Disk {mount} at {pct}%")
return issues
def check_docker():
"""Check container health via Docker API."""
issues = []
result = subprocess.run(
["docker", "ps", "--format", "{{.Names}}:{{.Status}}"],
capture_output=True, text=True
)
if result.returncode != 0:
return ["[ALERT] Docker daemon not responding"]
for line in result.stdout.strip().split("\n"):
if not line:
continue
name, status = line.split(":", 1)
if "unhealthy" in status.lower():
issues.append(f"[ALERT] Container {name} is unhealthy")
elif "restarting" in status.lower():
issues.append(f"[ALERT] Container {name} is restarting")
elif "exited" in status.lower():
issues.append(f"[WARN] Container {name} exited")
return issues
def check_memory():
"""Parse /proc/meminfo for available memory."""
issues = []
with open("/proc/meminfo") as f:
meminfo = {}
for line in f:
key, val = line.split(":", 1)
meminfo[key.strip()] = int(val.strip().split()[0])
available_mb = meminfo.get("MemAvailable", 0) // 1024
if available_mb < THRESHOLDS["mem_min_mb"]:
issues.append(
f"[ALERT] Low memory: {available_mb}MB available"
)
return issues
def check_load():
"""Check 15-minute load average."""
issues = []
with open("/proc/loadavg") as f:
load_15 = float(f.read().split()[2])
if load_15 > THRESHOLDS["load_max"]:
issues.append(
f"[ALERT] Load average (15min): {load_15:.2f}"
)
return issues
def send_telegram(messages):
"""Send alerts to Telegram chat."""
text = "\n".join(messages)
url = f"https://api.telegram.org/bot{TELEGRAM_TOKEN}/sendMessage"
requests.post(url, json={
"chat_id": CHAT_ID,
"text": f"🖥 Server Check\n\n{text}",
"parse_mode": "Markdown"
})
if __name__ == "__main__":
all_issues = []
all_issues.extend(check_disk())
all_issues.extend(check_docker())
all_issues.extend(check_memory())
all_issues.extend(check_load())
if all_issues:
send_telegram(all_issues)
# No issues = no message. Silence is the feature.
The Cron Entry
One line in root's crontab:
# Server health check every 15 minutes
*/15 * * * * /usr/local/bin/healthcheck.py >> /var/log/healthcheck.log 2>&1That is the whole thing. If my phone is quiet, everything is fine. If it buzzes, something needs attention.
Why Not Prometheus / Grafana
I already had Prometheus and Grafana running. I still do. They are great for looking at trends, historical data, capacity planning.
But for "is something broken right now" they are overkill. I do not want to configure Alertmanager, write routing rules, set up notification templates, and maintain another three containers just to know if a disk is full.
A 100-line Python script does the job. It has zero dependencies beyond requests, which I already had installed. If Telegram is down, I have bigger problems anyway.
What I Added Later
Three small things over the past year:
# SSL cert expiry check ( added after a cert expired on me )
def check_ssl():
import ssl, socket
issues = []
domains = ["davideandreazzini.co.uk", "git.davideandreazzini.co.uk"]
for domain in domains:
ctx = ssl.create_default_context()
with ctx.wrap_socket(socket.socket(), server_hostname=domain) as s:
s.settimeout(10)
s.connect((domain, 443))
cert = s.getpeercert()
expiry = cert["notAfter"]
# Parse and check days remaining
from datetime import datetime
exp_date = datetime.strptime(expiry, "%b %d %H:%M:%S %Y %Z")
days_left = (exp_date - datetime.utcnow()).days
if days_left < 14:
issues.append(f"[WARN] SSL cert for {domain} expires in {days_left} days")
return issues2. A simple heartbeat — if the script itself stops running ( server off, cron broken ), a separatecron job on a different machine notices and alerts me.
# On a separate machine, check if the healthcheck file was
# updated recently
*/30 * * * * find /mnt/nas/healthcheck -mmin -30 || \
curl -s "https://api.telegram.org/bot$TOKEN/sendMessage" \
-d "chat_id=$CHAT_ID" \
-d "text=[ALERT] Healthcheck script not running!"3. Log rotation for the output file, because I forgot about it for 6 months and the log got to 400MB.
Conclusion
The script is ugly. It uses subprocess to call df instead of parsing /proc/mounts. It does not have tests. It does not have type hints.
It has been running for 14 months without a single modification beyond adding the SSL check. It caught a full disk twice, a crashed container three times, and a memory leak in my Go app once.
Sometimes the boring solution is the right one.