Skip to content

/spawn + /new Workflow Audit, Findings, 2026-06-11

URL: https://mkdocs.justinsforge.com/memory/handoffs/spawn-new-workflow-audit-findings-2026-06-11/

Audit of the remote-worker spawning stack (/spawn, /new, forge_spawn_session.sh, forge_spawn_pool.sh, /sessions). Recommend-only pass; no production code changed. Two empirical tests were run against throwaway haiku TUIs (both cleaned up). Headline: the typing race is root-caused (with a reproducible C-u bug underneath it), and Claude Code 2.1.173 now ships a --remote-control CLI flag plus a positional prompt argument that makes the entire send-keys prompt dance obsolete.

1. What we're achieving

Spawn Claude Code sessions on Console as remote workers, drivable from anywhere (CLI, claude.ai/code Remote Control, Telegram remote-bridge). Requirements: fast startup (warm pool ~1s, cold 30-60s), reliable first-prompt delivery, correct model, a working bridge URL, clean lifecycle (no orphans, no poisoned sockets, no cgroup death on bridge restart).

Current architecture: tmux -L spawn-<name> per-spawn socket, systemd-run --user --scope cgroup escape, then a screen-scraping pipeline: send-keys the claude command, poll for ❯, send-keys /remote-control, poll for the banner, sleep 4s, send-keys the prompt literal, verify by capture-pane grep (prefix or paste chip), C-u-and-retype up to 3x on failure, send Enter, verify thinking indicator, then extract the bridge URL from the newest ~/.claude/sessions/*.json under 90s old.

2. What works well (keep)

  • Per-spawn sockets + pre-flight sweep. Blast-radius isolation is proven; the two-strike sweep with homebase protection is solid.
  • Cgroup escape via systemd-run --user --scope. Right design; bridge restarts no longer mass-kill workers (loose ends in section 5).
  • Warm pool design. Bridgeless pre-warm (SPAWN_NO_REMOTE=1), claim-time activation, flock discipline with the fd-close fix, pane_current_command liveness. All correct and battle-tested.
  • Idempotent re-spawn (existing session short-circuit) and cold-spawn fallback in claim, so /new never hard-fails.
  • Poll-don't-sleep for ❯ and the banner, 90s boot timeout, capture-pane on abort.
  • One-message-at-boot rule and handoff-file briefings.

3. Failure modes and root causes

3a. The prompt-typing race (today's incident), root cause found

Reproduced the mechanics on a throwaway TUI (claude 2.1.173, haiku, tmux -L audit-test). Two stacked defects:

Defect 1, the trigger: verification races the paint, not the buffer. capture-pane reads the rendered screen, not the input buffer. On a loaded Console (MCP init storms, heavy fleet), the Ink event loop lags; at capture time (+2s) the typed text can be received but not yet painted. The script cannot distinguish "not received" from "not yet painted", so it concludes the buffer is empty. Every refinement so far (sleep 4, prefix grep, paste-chip grep) only narrows this window; none can close it, because screen-scraping has no ground truth.

Defect 2, the amplifier: C-u does not clear the input box. Empirically verified: with a 150-char prompt in the box (it renders as literal wrapped text, no paste chip, two display rows), sending C-u deletes only back to the start of the current wrapped display row. The first row (~76 chars) survives, and the status bar shows "Ctrl+Y to paste deleted text" confirming a partial kill. So the retry loop's "safe clear" leaves a fragment, the retype appends a full copy onto it, and verification keeps racing under load: fragment + copy + copy = the 3x stack Justin saw this morning, with typed_ok never settling, so Enter never fired. Worse: if verification happens to pass mid-stack, the script would submit a corrupted prompt (fragment + full copy) and report success.

Net: any prompt longer than one display row (roughly >76 chars, which is nearly every real spawn prompt) makes the C-u retry path destructive. The 2026-06-08 paste-chip refinement fixed the chip false-negative but the underlying clear primitive was never safe.

3b. Other failure modes (known, mostly mitigated)

  • Banner text drift across Claude Code versions broke /new once; the loose regex is a treadmill, every TUI string is unstable API.
  • URL extraction heuristic ("newest sessions JSON under 90s, not my pid") can mis-attribute under concurrent spawns; the pool's activate_remote needs a snapshot-diff workaround for exactly this. Note: ~/.claude/sessions/*.json files are named by the claude process pid, so deterministic attribution is possible (pane_pid, then child claude pid, then sessions/<pid>.json).
  • /sessions quick still uses the pre-lessons pattern: chained send-keys "$prompt" Enter, fixed sleeps, no verification, bare tmux on the default socket. It violates four of the spawn skill's own lessons and the single-source-of-truth rule.
  • Cgroup migration loose ends (from project memory, 2026-05-04): no end-to-end smoke test, /resume still uses bare tmux new-session (would land back in the bridge cgroup), coordinator-bot KillMode unaudited.
  • Idle warm spares self-exit ~2h; handled by 60min recycle + liveness, fine as long as POOL_MAX_AGE stays well under 2h.

4. First-party alternatives (researched 2026-06-11)

claude --remote-control [name] flag + positional prompt argument, verified live on the installed binary (2.1.173). claude --help shows claude [options] [command] [prompt] and --remote-control [name] Start an interactive session with Remote Control (plus --remote-control-session-name-prefix). End-to-end test: claude --model <id> --dangerously-skip-permissions --remote-control rc-flag-test '<prompt>' booted, registered a bridge (/rc active in the status bar, bridgeSessionId in the sessions JSON), auto-submitted the argv prompt, and the model answered. Zero send-keys after launch. This is exactly the gap the community filed as claude-code#39347; it has since shipped. Official docs: Remote Control.

Implication: boot command, bridge activation, and prompt delivery all move from fragile keystroke injection into argv, which is atomic and cannot race a redraw. The TUI scraping that remains (❯ poll) becomes a pure health check, never a control channel.

/config "Enable Remote Control for all sessions": rejected. It would auto-bridge warm-pool spares, defeating the bridgeless-until-claimed design and cluttering the claude.ai/code list. The per-invocation flag is strictly better here.

Agent SDK / headless (claude -p, stream-json): keep for fire-and-forget, not a replacement for workers. Headless mode and the Agent SDK spawn sessions programmatically with structured I/O, but they do not register a Remote Control bridge, and the bridge UI is the product requirement (drive from phone). Forge already routes headless-appropriate work there (spawn-headless-agent, dispatcher pipe mode, /followup). No change recommended.

RemoteTrigger / claude.ai routines API: first-party scheduled cloud agents (/v1/code/triggers). Cloud execution, no Console-local tools, so not a spawn replacement; possibly interesting later as a /followup alternative for tasks that need no local state.

Verdict: Forge was fighting the tool; the tool caught up. The send-keys prompt dance was the right hack for its time and is now obsolete for prompt delivery. Migrate to argv; keep tmux (still needed for attach/inspect/kill, scrollback, socket isolation, cgroup scoping); keep the warm pool (the flag does nothing for the 30-60s claude+MCP cold boot, which is the pool's whole reason to exist).

5. Prioritized improvements

P0 (kills today's bug class) 1. scripts/forge_spawn_session.sh: launch with claude ... --remote-control "<name>" -- "$PROMPT_ARG" (printf %q quoting into the tmux start command), delete the /remote-control send-keys, the 4s settle, the entire type/verify/C-u/Enter block. Keep ❯//rc active polling as health checks and keep capture-on-abort. Keep SPAWN_NO_REMOTE boots flagless so spares stay bridgeless. 2. Same file: until item 1 lands (or anywhere send-keys typing survives), remove C-u as a clear primitive for anything that can wrap; it deletes one display row, not the buffer. The claim-time /remote-control literal in forge_spawn_pool.sh is short enough to be exempt.

P1 (correctness and convergence) 3. scripts/forge_spawn_session.sh + scripts/forge_spawn_pool.sh: deterministic URL extraction via pid (pane_pid, child claude pid, ~/.claude/sessions/<pid>.json); delete the newest-file heuristic and the pool's bridge-id snapshot-diff. 4. .claude/skills/sessions/SKILL.md quick subcommand: replace the inline tmux loop with a loop over forge_spawn_session.sh (single source of truth; it currently re-implements the racy 2026-04 pattern). 5. Close the 2026-05-04 cgroup loose ends: bridge-restart smoke test, /resume skill cgroup escape, coordinator-bot KillMode audit (files: .claude/skills/resume/, forge_telegram_inbox_brain.py unit).

P2 (polish) 6. Pass --remote-control <name> with the tmux session name so the claude.ai/code list shows the real worker name instead of "snug-dragon" (verify the name actually surfaces; the JSON field remoteControlSessionName read back null in the test). 7. forge_spawn_pool.sh: lean boot profile for spares (skip chrome/gmail MCP) to cut refill time; already noted in reference_spawn_pool. 8. Retire the banner-regex treadmill once item 1 lands (the flag makes banner detection optional). 9. Update .claude/skills/spawn/SKILL.md lessons list and memory/general/reference_spawn_pool.md after migration; the C-u lesson (this doc, section 3a) belongs there verbatim.

Cost note: all P0/P1 items are edits to existing scripts, no new infra, no new tokens at runtime (argv delivery removes ~10s of sleeps per spawn).

6. Open questions for Justin

  1. Migrate forge_spawn_session.sh to the --remote-control flag now (P0.1), or run a one-week parallel period with the old path behind a SPAWN_LEGACY=1 escape hatch? (Recommendation: straight cutover; the old path is the bug.)
  2. Does the --remote-control <name> display name matter to you in the claude.ai/code list? If yes, P2.6 gets promoted.
  3. /sessions quick is barely used; converge it on the canonical script (P1.4) or delete the subcommand?
  4. The minimum supported Claude Code version becomes 2.1.x (flag availability) for the spawn path; Console pins ~/.local/bin/claude, any reason to hold back upgrades?

Sources: local claude --help (2.1.173, the authority), Remote Control docs, claude-code#39347, Agent SDK overview, empirical TUI tests on Console 2026-06-11 (C-u row-kill repro; --remote-control + argv prompt end-to-end pass).

Addendum: fix shipped same day (P0 + pool repair), 2026-06-11

Justin said ship it; the cutover landed with three discoveries the morning audit missed:

  1. Flag- and slash-activated Remote Control no longer write ~/.claude/sessions/<pid>.json in 2.1.173, and the TUI no longer prints the URL. Every json-based URL extraction (spawn script, pool snapshot-diff, /rebridge registry read) has been running on luck. The only local surface for the bridge id is the debug log (--debug-file), which logs [remote-bridge] v2 session URL: https://api.anthropic.com/v1/code/sessions/cse_.... Note the id prefix changed from session_ to cse_.
  2. The pool's activate_remote banner regex predated the /rc active status-bar text, so every /new claim was failing activation and silently degrading to a cold spawn (resilience held, spares wasted).
  3. local a="$1" b="path/${a}" under set -u is a trap: expansions in a local list evaluate before earlier assignments in the same list take effect.

Shipped (all end-to-end tested with throwaway haiku spawns plus a real warm claim): - scripts/forge_spawn_session.sh: argv delivery (--remote-control <name> -- '<prompt>'), --debug-file /tmp/spawn-rc-<name>.log on every boot (warm included), typing/verify/C-u block deleted, URL from debug log with legacy json fallback, soft submission check. - scripts/forge_spawn_pool.sh: activate_remote reads the spare's own debug log (race-free attribution), banner regex accepts /rc active. Pool drained + refilled so all spares carry logs. Warm claim verified at 1.0s with working URL. - scripts/forge_tmux_socket_sweep.sh: reaps /tmp/spawn-rc-<name>.log alongside its dead socket. - .claude/skills/spawn/SKILL.md: steps + lessons rewritten (send-keys prompt typing is now a forbidden pattern).

Verification debt, one tap: session p0final_Haiku45 is alive at https://claude.ai/code/cse_01YYN7FEzt6UqmWgbaznuJkP; open it once to confirm the cse_ URL form resolves in the web UI, then kill the session. If it does not resolve, the by-name listing (sessions are now named via --remote-control <name>) is the interim path and the URL format needs one follow-up.

Incident note: during testing I wrongly attributed ~/.claude/sessions/2560321.json to a test spawn and deleted it; the file belonged to a live 06:22 session and was restored from recorded values (bridge id, pid, startedAt) minutes later. remoteControlSessionName was null in the original; the restored copy matches.

P1 items still open: /sessions quick convergence, cgroup loose ends (/resume, smoke test, coordinator KillMode). The P1 "pid-walk URL extraction" from section 5 is RETIRED, built on the now-false pid-named-json assumption.

[spawn-audit_Fable5 worker]