URL: https://mkdocs.justinsforge.com/memory/handoffs/forge-health-audit-2026-05-11/
Forge Health Audit, 2026-05-11¶
7-day fleet, bots, automation, and silent-failure scan. Overall verdict: the user-facing surface is healthy, but two background safety nets (the daily canary and the context-prefetch daemon) have been silently dead for 8+ days. The system most at risk is the system that watches the system.
Red¶
forge-canary.service, failing daily since 2026-05-04¶
The daily smoke test that is supposed to alert when spawn / bots / reaper break has itself been broken for 7 consecutive days. Each run dies before executing user code.
May 04 14:03:20 console (python3)[2899269]: forge-canary.service:
Failed to determine supplementary groups: Operation not permitted
May 04 14:03:20 console systemd[949627]: forge-canary.service: Failed with result 'exit-code'
[...identical pattern May 05, May 06, May 07...]
systemctl cat forge-canary.service returns "No files found." The unit launches under systemd[949627] (a per-user manager), not systemd[1], which means the unit file lives somewhere a User= mismatch can't reach (likely ~/.config/systemd/user/ but the timer expects system scope or vice-versa). The Operation not permitted on supplementary groups is the classic signature of a unit declared with User=justinwieb running inside a user-scoped manager that lacks CAP_SETGID.
Fix path:
1. find / -name forge-canary.service 2>/dev/null to locate the actual unit file
2. Move from user scope to system scope (/etc/systemd/system/) and remove the User= line, OR keep user scope and remove the supplementary-group setup from the unit
3. systemctl daemon-reload && systemctl start forge-canary.service to verify
4. Reference: reference_daily_canary.md
Why this is red, not yellow: the canary's job is to be the early-warning siren for everything else in this audit. While it's down, future silent failures of bots, reaper, or spawn will not page Justin. Every yellow item below was discovered by manual audit; the canary should have caught at least the context-prefetch death below within 24 hours.
Yellow¶
forge-context-prefetch.timer last fired 2026-05-03¶
● forge-context-prefetch.timer
Active: active (elapsed) since Sun 2026-05-03 08:40:19 CDT
Trigger: n/a
State "elapsed" with Trigger: n/a = systemd thinks the timer has run to completion and will never fire again. Service has run zero times in 8 days. Section 12 Daemon Layer cache for /recall daemon hits is therefore 8 days stale. User-visible impact small (semantic search still works against full index), but the lean prefetch layer is dark.
Fix: unit defines OnUnitActiveSec= but no OnCalendar= or OnBootSec=, so once the timer "elapses" once it stops. Check systemctl cat forge-context-prefetch.timer and add OnUnitInactiveSec=5min (or restart the timer: systemctl restart forge-context-prefetch.timer).
forge-general-purpose.service SIGKILL'd 2026-05-09 19:20¶
Manual systemctl restart triggered, stop-sigterm hit the 90s timeout, systemd had to SIGKILL the python parent plus two claude + node children. Came back clean, no functional impact, but the shutdown path does not propagate signals to the embedded Claude Code subprocess.
May 09 19:20:40 forge-general-purpose.service: State 'stop-sigterm' timed out. Killing.
[KILLs python 1092, claude 1908845, node 1908866, node 1908867]
Failed to kill control group [...]: Invalid argument
Fix: add KillMode=mixed and TimeoutStopSec=15s to the unit so SIGTERM hits the whole cgroup at once. Same fix likely applies to the other long-poll bots; verify on forge-remote-bridge and forge-lifeos-coordinator.
n8n webhook list-recent-emails returns "No item to return was found"¶
This is the legacy n8n Gmail-list workflow. Direct-client Google Calendar/Gmail cutover already happened (2026-04-30), so user-facing flows are unaffected, but any caller still pointing at this webhook will get a 500. Recommend either fixing the workflow or deleting it from n8n entirely to remove the dead surface.
750 dirty files in git working tree, no commit since the Phoenix sweep¶
git status shows 750 changes (mostly deletions from the Phoenix-era archive sweep). Auto-dream wrote LESSONS.md at 04:05 today, so writes are happening, but nothing has been committed to bookmark the cleanup. Risk: a future git stash or accidental git checkout . wipes a week of background dream output. Recommend a single bulk commit, "Phoenix sweep: archive retired files," to land the cleanup before the next session.
ssh-status skill has no run.sh¶
Skill at .claude/skills/ssh-status/ only ships SKILL.md. Anything that tries bash run.sh (boot-briefing health probe used to do this) gets "No such file or directory." Direct ssh loop works fine (verified, 9/9 hosts reachable). Cosmetic, but flag for cleanup.
Green¶
Fleet hosts, 9/9 reachable¶
| Host | Load avg (1/5/15) |
|---|---|
| finn | 0.49 / 0.43 / 0.41 |
| plex | 0.49 / 0.43 / 0.41 |
| media-server | 0.49 / 0.43 / 0.41 |
| n8n | 0.49 / 0.43 / 0.41 |
| frigate | 0.49 / 0.43 / 0.41 |
| adguard | 0.49 / 0.43 / 0.41 |
| immich | 0.49 / 0.43 / 0.41 |
| homeassistant | 0.07 / 0.02 / 0.00 |
(LXC containers report Finn host load via lxcfs, hence identical values across the LXC fleet; HA is a separate VM. Not a bug.)
Telegram bot fleet¶
All five bots active, zero restarts.
| Bot | State | Restarts | Last enter |
|---|---|---|---|
| forge-inbox-capture | active/running | 0 | 2026-05-03 08:40 |
| forge-lifeos-coordinator | active/running | 0 | 2026-05-04 18:49 |
| forge-notify-bot | active/running | 0 | 2026-05-03 08:40 |
| forge-remote-bridge | active/running | 0 | 2026-05-04 14:08 |
| forge-general-purpose | active/running | 0 | 2026-05-09 19:20 (after manual restart, see yellow) |
| forge-inbox-webhook | active/running | 0 | 2026-05-03 08:40 |
| forge-bot-watchdog.timer | active, firing every 5 min | n/a | 13:46:13 (this minute) |
No error-level entries in any bot journal over the last 7 days.
Dashboards and APIs¶
forge-dashboard: 200 OK (http://127.0.0.1:8099)
forge-usage: 200 OK (http://127.0.0.1:8098)
forge-context-api: 200 OK (http://127.0.0.1:7358)
mkdocs: 200 OK (http://127.0.0.1:8000)
Auto-dream and auto-memory¶
/home/justinwieb/forge/logs/auto-dream.log shows a clean run every day from 2026-05-02 through 2026-05-11. No gaps, no errors. Today's run: 0 merged, 0 stale, 0 promoted, 13 orphans (steady state).
Daily working logs present every day:
| Date | Lines | Bytes |
|---|---|---|
| 2026-05-05 | 12 | 528 |
| 2026-05-06 | 40 | 2 012 |
| 2026-05-07 | 152 | 6 327 |
| 2026-05-08 | 14 | 528 |
| 2026-05-09 | 116 | 5 107 |
| 2026-05-10 | 55 | 2 265 |
| 2026-05-11 | 13 | 547 (this session, growing) |
Timers, firing on schedule¶
| Timer | Last | Next |
|---|---|---|
| forge-bot-watchdog (5 min) | 13:41 | 13:46 |
| forge-followup-dispatcher (1 min) | 13:45 | 13:46 |
| forge-reap-orphan-claudes (15 min) | 13:33 | 13:48 |
| forge-tmux-socket-sweep (10 min) | 13:36 | 13:46 |
| forge-mkdocs-build (2 min) | 13:44 | 13:46 |
| forge-habits-instantiate (04:00 CT) | today 04:00 | tomorrow 04:00 |
| forge-habits-alias-backfill (03:30 CT) | today 03:30 | tomorrow 03:30 |
| forge-habits-streaks (23:55 CT) | last night 23:55, clean | tonight 23:55 |
| forge-time-daily (3x/day) | 12:00 | 17:00 |
| forge-telegram-hourly-checkin | 13:00 (pushed) | 14:00 |
| forge-telegram-followup-unscheduled (18:00) | yesterday | today 18:00 |
| forge-telegram-weekly-review-prep (Sun 17:30) | last Sun | next Sun |
Habits-streaks: a one-off exit-code failure on 2026-05-06 23:55 followed by clean runs every night since (May 7-10). Self-healed.
Hourly check-in: noted as failing on 2026-05-05 10:00, but pushed cleanly every hour on 2026-05-10 from 17:00 through 19:00+ and every hour 2026-05-11. Self-healed.
Dispatcher and task queue¶
forge-dispatcher.service active/running. tasks/pending/ empty, tasks/active/ empty, tasks/completed/ archived May 2. No backlog, no failed jobs in the last 7 days.
n8n container¶
Up 7 hours (post-reboot-ish), 20 workflows loaded including the v2 direct-client successors and legacy webhooks. Only error in the last 30 docker log lines is the list-recent-emails "no item" noted in yellow.
Eval harness and Lessons¶
LESSONS.md last touched 2026-05-11 04:05 (auto-dream writes it). No recent regression entries; latest section is the 2026-04-28 Phase 4.5 eval-harness wiring note. No new incidents logged since the Pure Phoenix 4.9 incident loop wired up.
Recommended order of operations¶
- Fix the canary first. Until it runs, the rest of this audit is invisible to the alerting layer. ~10 min, system-scope unit move.
- Restart context-prefetch timer with
OnUnitInactiveSec=added. ~5 min. - Add
KillMode=mixed+TimeoutStopSec=15sto the three long-poll bot units. ~5 min. - Single bulk commit of the 750 dirty Phoenix-cleanup files so the working tree resets to clean.
- Delete or fix the dead
list-recent-emailsn8n workflow. - Backfill
ssh-status/run.sh(or remove any caller that expects it).
None of the above is blocking. The user-facing surface (bots, dashboards, semantic search, scheduled jobs, fleet) is fully healthy as of 2026-05-11 13:45 CDT.
[Claude Code, forge-health-audit_Opus47]