Skip to content

URL: https://mkdocs.justinsforge.com/memory/general/reference_creatortrack_dev_slot_reaper/

CreatorTrack dev-slot reaper

Auto-teardown of idle CreatorTrack parallel-dev slots. Exists because on 2026-07-06 four abandoned dev slots (each a next dev worktree server, ~2GB RSS apiece, two next-server processes each) plus a morning compile storm drove Console to 100% CPU / 99% RAM and the kernel OOM-killer reaped a next-server, forcing a VM restart. See daily log 2026-07-06.

What it does

  • Script: scripts/forge_creatortrack_dev_slot_reaper.py (stdlib only).
  • Timer: forge-creatortrack-dev-slot-reaper.timer -> .service, every 15 min (first run 10 min after boot). Runs as justinwieb. Unit files mirrored in repo at infra/systemd/.
  • Lists worktrees under ~/forge-suite-wt/<branch>, reads each slot's port from its .env.local, and only considers slots whose server is actually listening (a dead worktree holds no RAM, left alone).

Idle signal (no false kills)

A slot is idle only when BOTH signals agree, so an actively-worked slot is never reaped:

  1. No browser tab: its dev-server port has zero ESTABLISHED TCP connections. An open browser tab holds a persistent HMR websocket, so a slot you are watching always has >=1 connection. Via ss -tnH state established '( sport = :PORT )' (source-port filter = local). Unknown/error -> treated as active (conservative).
  2. No recent edits: no source file edited within CT_REAPER_ACTIVE_MINUTES (default 90). Source = git ls-files --cached --others --exclude-standard (tracked + untracked-not-ignored), so the dev server's constant .next/ + node_modules/ churn and the generated .env.local do NOT count -- only real edits. This protects a slot being built through a chat session, which holds no browser connection but is actively edited (the 2026-07-08 data-loss case). git error -> treated as active.

Only after a slot is idle by BOTH for CT_REAPER_IDLE_MINUTES (or is over the hard cap) is it a reap candidate; even then a dirty worktree is preserved.

Reap triggers

  1. Grace: idle continuously for >= CT_REAPER_IDLE_MINUTES (default 45). Per-branch first-idle epoch tracked in data/creatortrack/dev_slot_reaper_state.json.
  2. Hard cap: if live slots > CT_REAPER_MAX_SLOTS (default 4), reap the longest-idle slots (idle ones only, oldest-idle first) back to the cap, ignoring grace. Never reaps a slot with active connections, even over cap.

Teardown ALWAYS goes through the canonical forge_creatortrack_agent_teardown.sh <branch> (server + worktree + ct_<branch> clone DB). Branch ref is preserved; main/master/base and real DBs are guarded. Notifies via forge_notify.sh on each reap/failure. Log at logs/creatortrack_dev_slot_reaper.log.

Uncommitted-work guard (2026-07-08, never delete in-progress code)

The RAM problem a slot causes is the next dev process, NOT the on-disk files. So a slot with uncommitted work is never deleted: the reaper checks git status --porcelain on the worktree, and if it is dirty (tracked changes OR untracked files) it calls teardown with --server-only, which stops the server (freeing the RAM, the whole point) but keeps the worktree + clone DB. Only a clean slot is fully removed. A preserved slot's server is down so it holds no RAM and drops out of the next run's live set; its files sit on disk until you commit/merge or tear it down deliberately. Restart it any time with forge_creatortrack_agent_bootstrap.sh <branch> <port> (the worktree already exists, so bootstrap just relaunches the server).

The guard lives in forge_creatortrack_agent_teardown.sh itself (protects every caller, not just the reaper): a dirty worktree auto-downgrades to server-only unless --force-dirty is passed. An unreadable git state is treated as dirty (fail safe: never risk deleting work on an unknown state). Preserved slots notify at warning (higher signal than a routine reap) so you know work was parked, not lost. Why: on 2026-07-08 a full task's worth of uncommitted code was wiped when a slot reaped mid-build. Lesson also in ct-dev-slot-commit-often (commit each task before the next).

Teardown flags: --server-only (stop server, keep everything), --force-dirty (allow full removal of a dirty tree), --keep-db (remove server + worktree, keep the clone DB).

Claimed-slot guard (2026-07-15, a clean tree does not mean a disposable DB)

A CLAIMED registry slot in memory bands 1-2 is never fully torn down, even with a clean worktree. The dirty-tree guard only vouches for CODE; the slot DATABASE can hold hours of un-recreatable work. On 2026-07-15 an overnight transcript scrape (546 YouTube caption pulls, rate-limited over ~6h) finished into ct_warm_slot_1, the dev server later died under memory pressure, the slot went idle 180m with a clean tree, and the band-2 reap dropped the DB, destroying the entire corpus (code survived; data did not). Now the reaper downgrades idle claimed slots to --server-only in bands 1-2 (RAM freed, worktree + DB kept, registry state stays claimed so the warm-pool maintainer never re-warms over it). Band 3 (>85%, OOM territory) keeps full-teardown authority. Discard a preserved claimed slot deliberately with forge_creatortrack_slot_claim.sh release <n>. Companion practice: any expensive-to-acquire data landing in a slot DB gets exported to a durable file under forge/data/ as soon as it is complete (the wipe also motivated data/creatortrack/social-transcripts-ws85.jsonl).

Ops

  • Manual dry-run: python3 scripts/forge_creatortrack_dev_slot_reaper.py --dry-run
  • Force now: sudo systemctl start forge-creatortrack-dev-slot-reaper.service
  • Tune: edit Environment=CT_REAPER_IDLE_MINUTES/MAX_SLOTS in the .service.
  • Related: dev-slot teardown, parallel dev agents.

Console capacity (raised same day)

Console VM (Finn 103) bumped 4 vCPU/50GB -> 8 vCPU/80GB (qm set 103 --cores 8 --memory 81920) to raise the compile-storm ceiling. Finn (i9-13900H, 128GB, 20 threads) has ample headroom. The reaper is the real fix; the resize is slack.