Plan: forge-monitor unified fleet monitor service

URL: https://mkdocs.justinsforge.com/memory/plans/forge-monitor-2026-06-09/

Date: 2026-06-09 Approved spec: Build scripts/forge_monitor.py, a procedural (zero-LLM) monitor run by forge-monitor.timer every 5 minutes that writes the three dead logs (logs/monitor-security.log, logs/monitor-infra.log, logs/monitor-business.log) in the exact format their two consumers already parse, so boot briefing and /fleet-status revive without modification. Alerting uses the canonical dedup + all-clear state machine (warn once, escalate sustained at 30 min, suppress repeats 60 min, always send recovery) via scripts/forge_notify.sh. Pre-approved by Justin 2026-06-09 per the build brief; plan written for the record per the brief's hard gate.

Consumer contract (verified by reading both consumers): - scripts/forge_fleet_status.sh collect_monitors(): health from log mtime (<15 min healthy, <60 stale, else dead); status forced to alert if last line contains ALERT or CRITICAL; last line truncated to 200 chars. Recent-events parser accepts bracketed [YYYY-MM-DD HH:MM:SS] timestamps. - scripts/forge_hook_session_start_since_last.sh: last line matching (CHECK:|: OK\b|SUCCESS:|RESOLVED:) is the all-clear marker; any later line matching (CRITICAL|ERROR|ALERT|FAIL) is shown as an unresolved alert; logs untouched >24 h are skipped. - Therefore per cycle, per domain, exactly one line is appended: - healthy: [YYYY-MM-DD HH:MM:SS] CHECK: <domain> OK (<n> checks) - failing: [ts] ALERT: <domain>: <detail> (or CRITICAL: once escalated) - first healthy cycle after an alert: [ts] RESOLVED: <domain>: <detail> (counts as all-clear in both consumers) - CHECK/RESOLVED lines must never contain the words ALERT/CRITICAL/ERROR/FAIL.

Checks: - infra: (1) Console systemctl --failed count; (2) ssh finn pct list vs data/monitor_expected_state.json (CT 110 minecraft expected stopped per reference_minecraft_server.md, all others running); (3) root-disk usage per host over the ssh-status alias list (finn plex media-server n8n frigate adguard homeassistant immich; minecraft skipped while CT 110 is stopped) with warn ≥85% / crit ≥93%, plus Console local; (4) NFS/FUSE mount health on Console: timeout-guarded stat of /mnt/workspace and any Finn-served mounts found in /proc/mounts; (5) commit hygiene: count of modified+untracked repo files older than 24 h (git status --porcelain + mtime), warn ≥1 through the same dedup machine. - security: (1) sshd auth-failure count from journal on Console and Finn over the last 15 min (journalctl -t sshd), warn >10 / crit >50; (2) systemctl --failed on Finn (failed-unit anomalies); (3) cloudflared process present on Console and Finn (pgrep). - business: HTTPS reachability (any response with status <500 within 10 s counts as up; CF Access 302s are fine) for finances, usage, tasks, hyperframes, mkdocs, dashboard, erpnext at .justinsforge.com.

State + alerting: data/forge-monitor-state.json, keyed per check-id: first_fail_ts, last_alert_ts, last_alert_level. First failed cycle → warning page; still failing after 1800 s → critical page; re-page same level only after 3600 s; recovery after any page → info RECOVERED page. Pages via scripts/forge_notify.sh <priority> <subject> <message>. Mirrors forge_monitor_mount_watchdog.sh, the canonical implementation.

Engineering constraints: stdlib-only python3; sequential subprocess calls (no threading near subprocess per feedback_no_threads_with_subprocess_run); per-call timeouts (ssh ConnectTimeout=5 + subprocess timeout, HTTP 10 s) so worst case ≈2 min < 5-min cadence; ZoneInfo("America/Chicago"), no date.today(); every except re-raises, notifies, or carries # Why swallowed:; a check that errors (e.g. ssh unreachable) is a failed check, not a swallowed one; idempotent (append-only logs, upsert state). Read-only toward Finn and LXCs.

Out of scope: - No changes to forge_fleet_status.sh, the boot-briefing hook, or any Telegram bot script. - No remediation actions (no restarts, no remounts); observe and page only. Mount repair stays with forge_monitor_mount_watchdog.sh. - No new public surfaces, dashboards, or LLM-driven analysis. - CT 110 stopped is normal, never flagged or auto-started.

Task list

Task 1: Create expected-state map

• Files: /home/justinwieb/forge/data/monitor_expected_state.json
• What: JSON map of Finn CTs to expected status from today's pct list (101,102,104,105,106,107,108 running; 110 stopped), plus disk thresholds and the host list, so policy lives in data not code. Force-add if data/ is gitignored (it is policy data, not runtime state).
• Verification: python3 -c "import json; d=json.load(open('/home/justinwieb/forge/data/monitor_expected_state.json')); assert d['containers']['110']=='stopped'; print('ok')"
• Commit: monitor: expected-state map for forge-monitor (CT 110 stopped is normal)

Task 2: Write forge_monitor.py

• Files: /home/justinwieb/forge/scripts/forge_monitor.py
• What: Single-pass monitor implementing the three check domains, log-line contract, and dedup/all-clear state machine above. --dry-run flag prints results without notifying (still writes logs unless --no-log).
• Verification: python3 /home/justinwieb/forge/scripts/forge_monitor.py --dry-run exits 0 and prints per-domain results; tail -1 of each logs/monitor-*.log shows a fresh bracketed line.
• Commit: monitor: forge_monitor.py, procedural fleet monitor reviving the 3 dead logs

Task 3: Install systemd service + timer

• Files: /home/justinwieb/forge/scripts/forge-monitor.service.unit, /home/justinwieb/forge/scripts/forge-monitor.timer.unit, copies at /etc/systemd/system/forge-monitor.{service,timer}
• What: Oneshot service running the script, timer OnBootSec=2min + OnUnitActiveSec=5min (no overlap with oneshot), system level like the other forge-* units; daemon-reload, enable --now the timer.
• Verification: systemctl list-timers forge-monitor.timer --no-pager shows NEXT within 5 min; systemctl status forge-monitor.service --no-pager shows last run exited 0.
• Commit: monitor: forge-monitor.service + .timer, 5-min cadence

Task 4: Prove consumers revive

• Files: none (verification only; fix regressions in forge_monitor.py if found)
• What: Run one real timer-driven cycle, then run both consumers and confirm the contract holds end to end.
• Verification: bash /home/justinwieb/forge/scripts/forge_fleet_status.sh 2>/dev/null | python3 -c "import json,sys; m=json.load(sys.stdin)['monitors']; print(m); assert all(x['status']=='healthy' for x in m)" and boot-briefing hook output shows no stale monitor alerts.
• Commit: none unless fixes needed (monitor: fix <issue> found in consumer verification)

Task 5: Register and document

• Files: /home/justinwieb/forge/memory/general/reference_forge_monitor.md (with mkdocs URL line), /home/justinwieb/.claude/projects/-home-justinwieb-forge/memory/MEMORY.md (index line; also retire/annotate project_dead_monitor_logs.md), /home/justinwieb/forge/memory/handoffs/forge-monitor-build-brief-2026-06-09.md (outcomes section)
• What: Topic file documenting checks, state file, log contract, units; MEMORY.md index entry; mark the dead-monitor-logs project memory resolved; append outcomes to the brief.
• Verification: bash /home/justinwieb/forge/scripts/forge_eval_run.sh passes.
• Commit: monitor: register forge-monitor (topic file, index, handoff outcomes)