URL: https://mkdocs.justinsforge.com/memory/general/reference_forge_monitor/

forge-monitor, unified fleet monitor

Procedural (zero-LLM) monitor that revived the three logs dead since the 2026-04-28 Phoenix refactor. Built 2026-06-09 per forge-monitor-build-brief-2026-06-09; plan at forge-monitor-2026-06-09.

Pieces

Piece	Path
Script	scripts/forge_monitor.py (stdlib-only python3, --dry-run, --no-log)
Units	scripts/forge-monitor.{service,timer}.unit, installed at /etc/systemd/system/forge-monitor.{service,timer} (system level, oneshot + OnUnitActiveSec=5min, OnFailure=forge-unit-failure-notify@%n)
Policy data	data/monitor_expected_state.json (tracked in git): expected CT states, disk/auth thresholds, business URLs, Console mounts, Finn data mounts (mounts_finn: /mnt/storage, /mnt/pve/workspace; added 2026-07-01 after /mnt/storage silently hit 97%)
Alert state	data/forge-monitor-state.json (gitignored runtime state)
Output	logs/monitor-security.log, logs/monitor-infra.log, logs/monitor-business.log

Log-line contract (consumers parse this; do not change)

Consumers: scripts/forge_hook_session_start_since_last.sh (boot briefing) and scripts/forge_fleet_status.sh (/fleet-status, mtime freshness: <15 min healthy).

• healthy: [YYYY-MM-DD HH:MM:SS] CHECK: <domain> OK (<n> checks)
• failing: [ts] ALERT: <domain>: <detail> (CRITICAL: once escalated)
• first healthy cycle after failure: [ts] RESOLVED: <domain>: all checks healthy again
• CHECK/RESOLVED lines must never contain ALERT/CRITICAL/ERROR/FAIL.

Checks (29 total as of 2026-07-10)

• infra (14): Console failed units; Finn pct list vs expected map (CT 110 minecraft never alerts in either state, see reference_minecraft_server); root disk on Console + Finn + 7 LXCs (warn >=85%); Finn data mounts; /mnt/workspace mount stat.
• security (5): sshd auth-failure counts from journal on Console + Finn (15-min window, warn >10); Finn failed units; cloudflared process on Console + Finn.
• business (7): HTTPS reachability (<500 within 10s counts as up, CF Access redirects fine) for usage, tasks, hyperframes, mkdocs, dashboard, erpnext, fitness (finances removed 2026-07-10, app retired).
• wellness (3): data-freshness tripwires for Eight Sleep, Garmin, Hevy.

commit_hygiene removed 2026-07-10: auto-memory and auto-dream keep the repo perpetually dirty, so the check sat red for weeks and paged noise. Justin explicitly does not want commit-hygiene pages; do not re-add without asking him.

One combined SSH round trip to Finn covers pct/disk/journal/units/cloudflared. Sequential subprocess calls only (no threads near subprocess). Read-only toward Finn and LXCs.

Alerting

Canonical dedup + all-clear state machine (feedback_alert_dedup_allclear_pattern.md), per check key: first fail pages warning; sustained >30 min escalates critical; same level re-pages only after 60 min; recovery after any page sends info RECOVERED. Pages via scripts/forge_notify.sh. Unit crash itself pages via forge-unit-failure-notify@.

Human paging layer (2026-07-10): humanize(key, detail, phase) in forge_monitor.py translates each check key into a plain-English Telegram page (what broke, why it matters, "Tap Spawn fixer"), with the terse machine detail kept as a trailing Details: line. Domain LOG lines are untouched (log-line contract above). container_names in monitor_expected_state.json maps CT ids to names so pages say "CT 113 (forge-openwebui)", keep it in sync when adding containers. New check keys without a template fall back to Fleet check failing: <key>, add a branch in humanize() when adding checks.

Ops

systemctl list-timers forge-monitor.timer
systemctl status forge-monitor.service
python3 scripts/forge_monitor.py --dry-run --no-log   # side-effect-free run

Tune thresholds and URL list in data/monitor_expected_state.json, no code edit needed. Mount repair stays with forge_monitor_mount_watchdog.sh; forge-monitor observes and pages only.

[Claude Code, forge-monitor build session 2026-06-09]