I’ve got a number of apps running on my homelab and got tired of addressing some routine alerts. So I built a small pipeline where the alert and the investigation arrive as one Telegram message: what fired, what an AI agent found when it actually ran ping / PromQL / Loki queries, and a proposed fix.
- The AI is never in the critical path. severity=critical routes to a contact point that hits Telegram directly (what I’ve been using for most alerts from home) AND the webhook, so pages behave exactly like they did before this project existed. Only warning level alerts wait for their diagnosis.
- The backstop poller pushes a heartbeat to Pushgateway, and a plain critical severity rule fires if it goes stale. If the pipeline dies, the path that does not depend on it tells me.
- Push for speed, poll as backstop. Webhook retries are finite, so a 15 minute poller catches anything a down receiver missed. Both share one flock guarded fingerprint state, so nothing double notifies.
- No auto remediation, on purpose. If a finding is benign but the rule is noisy, the investigator has to propose a rule tuning (longer
for:, threshold, deletion) instead of shrugging.
The investigator is any CLI that takes a prompt and can run shell commands. I use an OpenClaw agent (mixed feelings but it works fairly well); claude -p works too.