Skip to content

forge-monitor build brief, 2026-06-09

Worker brief: build the unified fleet monitor service that replaces the three dead monitor logs. Approved by Justin 2026-06-09 ("do 1 and 2", item 2 of the audit follow-up menu).

The problem

logs/monitor-security.log, logs/monitor-infra.log, logs/monitor-business.log have had no writer since the Phoenix refactor on 2026-04-28, but two consumers still tail them and present stale "all clear" data every session:

  • boot briefing: scripts/forge_hook_session_start_since_last.sh
  • fleet status: scripts/forge_fleet_status.sh (surfaced by /fleet-status)

/fleet-status currently reports all three monitors as dead, last_run_minutes_ago: ~55000. See project_dead_monitor_logs.md in the Claude memory dir and section 2 of forge-audit-2026-06-09.

Requirements (hard)

  1. Procedural only, zero LLM calls. This runs on a timer forever; per feedback_subscription_over_api_for_automation.md it must cost $0 quota.
  2. Naming: forge-monitor.service + forge-monitor.timer (singleton, kebab-case, Section 3 taxonomy). Script at scripts/forge_monitor.py, flat-dir verbose-name rule.
  3. Write the three existing log files (same paths, same [YYYY-MM-DD HH:MM:SS] CHECK: ... line shape the consumers already parse) so boot briefing and fleet-status come back to life without modification. Verify the exact expected format by reading both consumers first.
  4. Checks, minimum set:
  5. infra: systemd --failed count on Console; pct list on Finn vs an expected-state map in data/monitor_expected_state.json (CT 110 minecraft expected=stopped, policy per reference_minecraft_server.md; all others expected=running); disk usage thresholds per host via existing ssh aliases (see /ssh-status skill internals at .claude/skills/ssh-status/); FUSE mount health (reuse logic or call scripts/ mount-check internals).
  6. security: failed-unit anomalies, SSH auth failures count from journal on Finn + Console, CF tunnel process health.
  7. business: HTTP 200 checks on the public surfaces (finances, usage, tasks, hyperframes, mkdocs, dashboard subdomains; list in system-map/fleet.md + CF tunnel configs), ERPNext reachability.
  8. Alerting: use the alert-dedup + all-clear state machine pattern from feedback_alert_dedup_allclear_pattern.md (warn once, suppress repeats, escalate sustained, always send recovery). Notify via scripts/forge_notify.sh warning|critical. State file in data/ (gitignored area is fine).
  9. Commit-hygiene check (item 3 of the menu, folded in): count untracked + modified files in the repo older than 24h (use git status --porcelain + file mtimes); warn through the same dedup machine. Ignore scripts/forge_finances_app.py style in-flight churn by requiring the 24h age.
  10. Idempotent, fail-loud per doctrine Section 9.5. No catch-and-ignore; every except re-raises, notifies, or carries # Why swallowed:.
  11. Timer cadence: every 5 minutes (matches what the old monitors claimed).
  12. Tz-aware local time only; date.today() is banned (feedback_date_today_banned_in_forge_scripts.md).

Process (hard gate)

Run /feature-plan FIRST and write the spec to memory/plans/forge-monitor-2026-06-09.md. Justin may review via Remote Control; proceed after writing the plan (he pre-approved the build, the plan is for the record and for catching contract mistakes with the two consumers). Then implement, install the unit + timer (system level, like the other forge-* units), run one real cycle, show the three log files populating and /fleet-status turning green, then commit and register: topic file memory/general/reference_forge_monitor.md + MEMORY.md index line + this handoff updated with outcomes.

Boundaries

  • Do not touch the Telegram bot scripts (another worker owns them today).
  • Do not modify scripts/forge_fleet_status.sh or the boot-briefing hook unless the log format genuinely cannot be matched; if so, document why in the plan first.
  • Read-only toward Finn and the LXCs (status checks via existing ssh aliases only).

[Claude Code, forge-audit fix session]

Outcomes (build session, 2026-06-09 14:20 CDT)

Shipped end to end, all requirements met:

  • Plan written per the hard gate: forge-monitor-2026-06-09; both consumers read first, log-line contract documented there.
  • scripts/forge_monitor.py (stdlib-only, zero LLM), 25 checks: infra 13, security 5, business 7. Commit-hygiene check folded in. One combined SSH round trip to Finn; sequential subprocess calls; tz-aware local time.
  • Dedup + all-clear state machine verified in isolation (6 scenarios: warn once, suppress, escalate sustained, re-page after 60 min, recover, CRITICAL log word) and state stored at data/forge-monitor-state.json (gitignored). Policy data at data/monitor_expected_state.json (tracked; CT 110 never alerts in either state per minecraft reference).
  • forge-monitor.service + forge-monitor.timer installed system-level, 5-min cadence, OnFailure=forge-unit-failure-notify@%n. First timer-driven run exited 0.
  • Verified live: all three logs writing [ts] CHECK: <domain> OK (n checks); /fleet-status reports all three monitors healthy (was dead, ~55000 min); boot briefing shows no stale alerts. Neither consumer was modified.
  • Registered: reference_forge_monitor + MEMORY.md index line; project_dead_monitor_logs.md memory retired as resolved.

[Claude Code, forge-monitor build session]