Skip to content

Forge System Audit, 2026-06-09

Comprehensive read-only audit of /home/justinwieb/forge. Grounded in system-map/{fleet,architecture,steering}.md, MEMORY.md, recent memory/handoffs/, LESSONS.md, plus live checks: ssh-status, systemd service/timer state, eval-harness last-full run, git working tree, AutoAgent state. No changes made.

Auditor: Claude Code (Opus 4.8), session forge-audit_Opus48.

Executive Summary

  1. The fleet is healthy: 22 forge-* systemd services active, 0 failed; all LXC/VM hosts green on disk and memory. The only host down is minecraft CT 110 (stopped, likely intentional).
  2. The single biggest risk is git hygiene: 101 untracked + 28 modified files, including entire shipped subsystems (autoagent/, comms/, dashboard/, clients/, usage-dashboard/, parts of data/) and 60 memory files never committed. This violates the steering doctrine that "all memory that matters lives in forge (git-tracked)" and risks loss on any stash/worktree operation.
  3. The memory pipeline is silently degrading: MEMORY.md is over both caps (236 lines / 32,750 bytes vs 200 / 25,000) and has failed eval nightly for ~6 weeks; auto-proposed/ has 111 staged proposals that auto-dream never promotes.
  4. LESSONS.md is drowning in noise: of 394 tagged entries, 363 are routine auto-memory / auto-dream / eval-failure logs and only ~15 are real incidents, defeating the file's purpose as an incident ledger.
  5. The platform has grown fast and well (AutoAgent with 7 projects, finances hub, datalayer, hyperframes/remotion hubs, time + habits tracking) but the supporting hygiene (gitignore, commit discipline, index pruning, lessons rotation) has not kept pace.

1. What's Working Well

Prioritized by how load-bearing each is.

Area State Evidence
Service fleet All 22 forge-* services active (running), 0 failed systemctl --failed empty; bots, dispatcher, context-api, dashboards, mkdocs, browser stack all up
Host health Every LXC/VM green ssh-status: finn 14G/94G (16%), media-server 31%, immich 56%, n8n 58%; memory within limits; 6d uptime
Eval harness Running daily, ran 03:00 today logs/eval-harness-last-full.json ts 2026-06-09T03:00:01; 9 of 13 checks pass
Em-dash boundary sanitize Working as designed no-em-dashes eval actual = 0; the deterministic forge_text_sanitize.strip_em_dashes filter holds where the prompt-only rule failed
Bot resilience Watchdog + canary live forge-bot-watchdog.timer (~5 min), forge-canary daily smoke test, orphan-claude reaper every 15 min
Spawn warm pool Shipped + healthy forge_spawn_pool.sh + forge-spawn-pool.timer; /new claims a pre-booted Opus in ~1s (fixed banner-match bug, commit 314d9de)
New product surfaces Live and serving AutoAgent (7 projects), finances.justinsforge.com, datalayer, hyperframes, remotion, tasks-checklist, usage + fleet dashboards all behind CF Access
Observability tooling ssh-status + fleet-status both run clean used directly in this audit

The architecture (Manager creates tasks, Dispatcher spawns workers, Monitors are zero-token bash, steering deny-list pre-flights every worker) is sound and the deny-list in scripts/forge_steering_intercept.py is a genuine safety asset.

2. What Can Be Improved

Prioritized P0 (highest impact) to P3.

P0, Git hygiene and uncommitted working tree

  • git status: 101 untracked + 28 modified. Untracked by top dir: memory (60), scripts (16), data (8), sites (7), .claude (2), plus single-dir whole subsystems autoagent/, comms/, dashboard/, clients/, usage-dashboard/, .venv-finance/, personal/, tasks/.
  • Core working scripts sit modified-but-uncommitted: scripts/forge_telegram_brain.py, forge_morning_report.py, forge_orient.py, forge_telegram_remote_bridge.py, and system-map files.
  • Why it matters: the doctrine (system-map/steering.md) states memory must be git-tracked. LESSONS.md already records a "git stash empty-dir wipe" incident. With this much uncommitted surface, one stash/worktree op can vaporize weeks of memory and shipped code. The most recent commits are all finance: work; the rest of the tree has drifted uncommitted.
  • Fix path: stage and commit memory + the new service dirs; add a .gitignore for venvs and ephemeral caches (see Quick Fixes).

P1, MEMORY.md over caps, failing eval for weeks

  • memory-md-line-cap actual 235 (cap 200) and memory-md-byte-cap actual 32,607 (cap 25,000) have both failed in the nightly full run; these tags recur 38 and 41 times in LESSONS.md.
  • The index has accreted detail that belongs in topic files. It is an error-severity check failing every single night with no remediation.
  • Fix path: prune MEMORY.md back under 200 lines / 25 KB, moving detail to memory/general/reference_*.md topic files (the index is meant to be one line per entry).

P1, auto-dream consolidation stalled

  • auto-proposed/ holds 111 staged memory proposals. Auto-dream's historical "Promoted from staging" count is 0 to 1 per run, so the staging area only grows.
  • Either the promotion gate is too strict or nobody reviews staging. Net effect: the auto-memory pipeline captures but never consolidates, so MEMORY.md and topic files do not benefit and the staging dir is dead weight.
  • Fix path: add a staging reaper (auto-archive proposals older than N days) or a weekly review nudge; investigate why forge_memory_auto_dream.py promotion threshold never fires.

P2, mkdocs-url-on-new-md warning, chronic

  • 23 MD files in indexed dirs missing the URL: header (recurs 42 times in LESSONS). Warning-severity, so it never blocks, so it never gets fixed.
  • Fix path: one-shot sweep to append the URL line; consider a pre-commit auto-inserter so new MD files self-comply.

P2, LESSONS.md signal-to-noise collapse

  • 394 entries: 156 auto-memory, 44 auto-dream, 163 recurring eval-failure logs, ~15 real incidents. The file is 5,766 lines. Its stated purpose (incident + doctrine-exception ledger) is buried.
  • Fix path: rotate routine auto-memory/auto-dream/eval-pass entries to a logs/ archive or separate file; keep LESSONS.md for genuine incidents and doctrine exceptions only.

P3, minecraft CT 110 stopped

  • pct status 110 = stopped; ssh-status shows UNREACHABLE. Fleet map lists it as a public WiebCraft server.
  • RESOLVED 2026-06-09: intentional. Justin's policy is WiebCraft stays off unless he turns it on to play; onboot: 0 is correct. See reference_minecraft_server.md.

P3, deleted .gitkeep protectors

  • git status shows D tasks/completed/.gitkeep and D tasks/failed/.gitkeep deleted. Per the LESSONS "git stash empty-dir wipe" entry, these .gitkeep files exist specifically to stop dir-emptying wipes. Restoring them is protective.

3. New Services To Add

Prioritized by value.

  1. Commit-hygiene guardrail (high value). A timer or Stop-hook that warns when untracked memory files or whole new dirs sit uncommitted for >24h, pushing a notify. Directly addresses the P0 risk. Pairs with a proper .gitignore.
  2. MEMORY.md auto-pruner (high value). Script that detects when the index exceeds 90% of cap and either moves the longest entries to topic files or flags them for review. Turns a chronic nightly eval failure into a self-healing loop.
  3. auto-proposed staging reaper (medium-high). Auto-archive or auto-promote staged proposals on a cadence so the 111-file pile cannot recur. Closes the auto-dream gap.
  4. LESSONS.md rotation job (medium). Nightly move of routine [auto-memory]/[auto-dream]/passing-eval blocks to logs/lessons-archive-YYYY.md, keeping the live file as a real incident ledger.
  5. Container auto-start/health watch (medium). Extend the existing monitor pattern to alert (and optionally pct start) when a fleet container that should be up (minecraft, others) is found stopped.
  6. .gitignore audit + secret-leak pre-commit (medium). .venv-finance/ and similar venvs are currently untracked-but-visible; a maintained .gitignore plus the existing steering deny-list extended to a pre-commit secret scan hardens the repo.

4. Quick Fixes (low effort, high value)

Exact paths and actions. None applied (read-only audit).

# Action Path / command
1 Add a .gitignore covering venvs and ephemeral caches so they stop showing as untracked /home/justinwieb/forge/.gitignore add: .venv-finance/, **/__pycache__/, data/spawn_pool/, data/context-prefetch.json
2 Restore the protective gitkeeps deleted from the tree git checkout -- tasks/completed/.gitkeep tasks/failed/.gitkeep
3 Commit the 60 untracked memory files (daily logs, topic files, handoffs) git add memory/ && git commit (honors "memory is git-tracked" doctrine)
4 Commit shipped-but-untracked subsystems git add autoagent/ comms/ dashboard/ usage-dashboard/ clients/README.md data/README.md
5 Prune MEMORY.md under 200 lines / 25 KB ~/.claude/projects/-home-justinwieb-forge/memory/MEMORY.md (move detail to memory/general/ topic files)
6 Append URL: line to the 23 flagged MD files list them with python3 scripts/forge_eval_harness.py --check mkdocs-url-on-new-md --list (or grep the check), then append the header
7 Triage the 111 staged proposals ~/.claude/projects/-home-justinwieb-forge/memory/auto-proposed/ review + promote or archive
8 Confirm minecraft intent ssh finn 'pct start 110' if it should be public again, else leave stopped

Fixes Applied (same day, 2026-06-09 fix pass)

All quick fixes plus the P0/P1 root causes were applied the same afternoon (commits 110bf7b..28b85dd):

  1. Eval caps were the bug, not MEMORY.md. eval.json enforced 200 lines / 25,000 bytes while FORGE-DOCTRINE.md Section 12 (and MEMORY.md's own header) say 300 / 35,000. Aligned eval.json to doctrine; that mismatch was the source of 6 weeks of nightly failures.
  2. MEMORY.md rebuilt: 267 lines / 31.5KB, under doctrine caps. Fixed 9 broken index links (bare names and ../../../ paths now forge: prefixed), indexed 27 previously orphaned topic files, archived 4 stale/duplicate ones, removed em dashes. memory-index-no-orphans now passes.
  3. Auto-dream root causes fixed (scripts/forge_memory_auto_dream.py): rejected staging proposals are archived to _archive/auto-rejected-<date>/ instead of being re-evaluated by Sonnet nightly forever; promotions now append an index line to MEMORY.md (this was why 31 orphans accumulated); forge: links resolve in orphan detection; date.today() replaced with tz-aware local date. Tested end-to-end with canned decisions. The 111-file staging pile drains on tonight's 04:00 run.
  4. LESSONS.md rotated: 374 routine auto-memory/auto-dream/eval-repeat blocks moved to logs/lessons-archive-2026-H1.md; 26 real incident and phase blocks remain (5,766 lines down to 660).
  5. URL headers: appended to all 24 flagged MD files; mkdocs-url-on-new-md now 0.
  6. Git hygiene: .gitignore covers .venv*/, data/{spawn_pool,audio,datalayer,ha-backups}, data/mam_seed_state.json; tasks/*/.gitkeep restored; entire drifted tree committed in 5 logical commits. Working tree clean.
  7. Bonus security fix: Context API bearer token was hardcoded in 4 scripts (forge_morning_report.py, forge_orient.py, forge_telegram_inbox_brain.py, forge_heartbeat_random.py); now loaded from ~/.forge-secrets/context-api.env (chmod 600). Capture + coordinator bots restarted and verified active on the new code path. Token unrotated (localhost-only API); rotate if paranoid.
  8. Eval harness full run: 13 of 13 checks pass.

Resolved: minecraft CT 110 stopped is the NORMAL state. Justin confirmed (2026-06-09) that WiebCraft stays off unless he turns it on to play; onboot: 0 is intentional. Recorded in reference_minecraft_server.md; future audits should not flag it.

Appendix: Live Check Data

  • Services: 22 forge-* active, 0 failed (systemctl --failed empty).
  • ssh-status: finn load 8.05 disk 16%; plex 36%; media-server 31%; n8n 58%; frigate 55%; adguard 36%; immich 56%; homeassistant 50%; minecraft UNREACHABLE/stopped.
  • Eval last-full (2026-06-09T03:00:01): PASS (9): doctrine-present, no-em-dashes (0), doctrine-mkdocs-serves, no-persona-names-in-code, naming-taxonomy-services, lessons-md-schema-conformance, lessons-orphan-recurrence, no-pipefail-head-sigpipe, index-no-stale-status. FAIL (4): memory-md-line-cap (235), memory-md-byte-cap (32,607), mkdocs-url-on-new-md (23), memory-index-no-orphans (warning, 38 recurrences in LESSONS).
  • Git: 101 untracked, 28 modified; last commit 10d624c (finance, 4 min ago); memory/daily last committed 3h ago.
  • AutoAgent projects: gus-the-bass (running), sip-n-serve, creator-hub, affiliate-store, games-justinwiebelhaus, austin-things-to-do-v2, austin-things-to-do.
  • MEMORY.md: 236 lines / 32,750 bytes. auto-proposed staged: 111.
  • LESSONS.md: 5,766 lines, 394 tagged entries (156 auto-memory, 44 auto-dream, ~163 eval-failure, ~15 real incidents).