Skip to content

Plan: brain latency rewire, persistent brain server

URL: https://mkdocs.justinsforge.com/memory/plans/brain-latency-rewire-2026-06-09/

Date: 2026-06-09 Approved spec: Pre-approved as item 5 of the 2026-06-09 batch (brief). A persistent daemon (scripts/forge_brain_server.py, systemd unit forge-brain-server.service) holds warm Claude Agent SDK clients (one per model) and serves brain calls over 127.0.0.1 HTTP. The two brain scripts gain an engine dispatch controlled by FORGE_BRAIN_ENGINE (subprocess default, server, or persona-scoped server:capture); on any server error the call falls back to the existing claude -p subprocess path automatically. Staged rollout: capture first, soak, then coordinator + general. Out of scope: poller scripts, forge_telegram_format.py, callbacks, the bot tool layer (shipped today), forge_gmail.py, forge_email_triage.py, forge_finance_query.py, forge_monitor.py, forge_alert_fixer.py. No Anthropic API account, ever.

PROBE results (2026-06-09, measured)

Real capture-brain prompt (39,005 chars), claude-sonnet-4-6, Console, no ANTHROPIC_API_KEY anywhere (env + ~/.forge-secrets/ checked):

Path Latency
Cold claude -p (n=2) 6.41s, 4.91s
Warm SDK connect (once) 1.03s
Warm SDK round-trips, max_turns=1 (n=4) 3.46, 4.00, 4.46, 4.55s

Findings: 1. Billing: SDK calls succeeded with zero API keys present on the machine, so they can only ride the CLI's subscription OAuth. Confirmed against SDK source: it spawns the CLI binary (_internal/transport/subprocess_cli.py), inheriting CLI auth. 2. AVX2 trap (real): claude-agent-sdk 0.2.95 ships a bundled CLI at claude_agent_sdk/_bundled/claude and prefers it over PATH. On Console that binary would SIGILL. ClaudeAgentOptions(cli_path=...) overrides; the server MUST pin cli_path=~/.local/bin/claude. 3. Tail anatomy (quota JSONL, May+June, n=247 brain calls): med 9.9s, p75 20.6s, p90 59.7s. 64 calls exceeded 20s; 53 of those SUCCEEDED at median 37s producing only ~734 chars, with prompt sizes equal to fast calls. Mechanism: claude -p runs with default CLI tool access and no turn cap (bypassPermissions per feedback_no_permission_deny_lists.md), so the model sometimes does agentic CLI-tool work before emitting the JSON. The warm SDK path with max_turns=1 was flat 3.5-4.6s across all runs. The server therefore caps turns, which removes the tail mechanism, in addition to skipping the ~2s Node cold boot. 4. Tokens: ResultMessage.usage returns real input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, plus num_turns. Quota rows will carry these in extra (chars fields kept so the aggregator is unchanged). 5. Sessions: ClaudeSDKClient.query(prompt, session_id=...) multiplexes independent sessions over one warm CLI process; a fresh session_id per request keeps brain calls stateless.

Verdict: <20s p90 is achievable (probe shows ~4-5s steady-state). Proceed.

Architecture

  • One asyncio HTTP daemon on 127.0.0.1:8771, endpoints GET /health and POST /brain ({prompt, persona, model, timeout_s} in; {ok, raw, latency_ms, num_turns, usage} out).
  • One ClaudeSDKClient per model (sonnet for capture/coordinator, opus for general), guarded by an asyncio.Lock (traffic is low; queuing beats interleaved-stream parsing). Fresh session_id per request. max_turns=1. Client recycled after 200 queries or on transport death (reconnect costs ~1s).
  • Bot-side dispatch lives ONLY inside _call_sonnet / _call_sonnet_once: if engine resolves to server for this persona, POST to the daemon; on connection error, timeout, or non-ok response, log a warning and fall through to the unchanged subprocess path. The nine robustness layers and mp.Process + flock concurrency model are untouched because the children just make a localhost HTTP call instead of (or before falling back to) subprocess.run.
  • Kill switch: FORGE_BRAIN_ENGINE read at call time. Values: subprocess (default), server:capture (stage 1), server (all personas). Flip is an env change on the poller units plus restart; systemctl --user stop forge-brain-server alone also degrades cleanly to subprocess.

Task list

Task 1: Create server venv and forge_brain_server.py

  • Files: /home/justinwieb/forge/.venv-brain-server/ (new venv), /home/justinwieb/forge/scripts/forge_brain_server.py
  • What: Promote the probe venv to .venv-brain-server (claude-agent-sdk pinned). Write the daemon: warm per-model clients, lock-serialized query(), health endpoint, client recycle, fail-loud logging.
  • Verification: /home/justinwieb/forge/.venv-brain-server/bin/python scripts/forge_brain_server.py --selftest (starts server, hits /health, runs one real sonnet brain round-trip, prints latency, exits 0)
  • Commit: bots: forge_brain_server.py, warm Agent SDK daemon for brain calls

Task 2: systemd unit forge-brain-server.service

  • Files: ~/.config/systemd/user/forge-brain-server.service (plus a copy under /home/justinwieb/forge/infra/systemd/ if that convention exists)
  • What: User unit running the venv python, Restart=always, journal logging. Enable + start.
  • Verification: systemctl --user is-active forge-brain-server && curl -fsS http://127.0.0.1:8771/health
  • Commit: bots: forge-brain-server.service systemd unit

Task 3: Engine dispatch in forge_telegram_inbox_brain.py

  • Files: /home/justinwieb/forge/scripts/forge_telegram_inbox_brain.py
  • What: _call_sonnet gains _brain_engine() resolution and a _call_brain_server() helper (urllib, stdlib only); server errors fall back to the existing subprocess block unchanged. Default env keeps behavior byte-identical.
  • Verification: FORGE_BRAIN_ENGINE=server python3 -c one-shot calling _call_sonnet on a trivial prompt returns parsed JSON; then same with FORGE_BRAIN_ENGINE=subprocess.
  • Commit: bots: inbox brain engine dispatch with subprocess fallback

Task 4: Engine dispatch + real token quota in forge_telegram_brain.py

  • Files: /home/justinwieb/forge/scripts/forge_telegram_brain.py
  • What: Same dispatch in _call_sonnet_once, persona-scoped (server:capture matches persona). When served by the daemon, quota.record() keeps prompt/response chars but adds extra={"engine": "server", "tokens": {...}, "num_turns": N}.
  • Verification: one-shot _call_sonnet_once under both env values; tail -1 data/claude-quota/2026-06.jsonl shows the tokens block.
  • Commit: bots: persona brain engine dispatch, real token counts in quota rows

Task 5: Flip capture to server engine, live soak start

  • Files: capture poller systemd unit drop-in (exact unit name confirmed on the box)
  • What: Set Environment=FORGE_BRAIN_ENGINE=server:capture on the capture poller only, restart it. Coordinator/general stay on subprocess.
  • Verification: /bots-health green; next live capture exchanges appear in quota JSONL with engine: server and latency under 20s.
  • Commit: bots: capture poller rides forge-brain-server (stage 1)

Task 6: Register and document

  • Files: ~/.claude/.../memory/MEMORY.md index, memory/general/reference_brain_server.md, memory/handoffs/brain-latency-rewire-brief-2026-06-09.md (outcomes section)
  • What: Topic file with architecture, kill-switch/flip commands, probe numbers; MEMORY.md one-liner; handoff outcome update incl. the documented full-flip command. Run bash scripts/forge_eval_run.sh.
  • Verification: bash /home/justinwieb/forge/scripts/forge_eval_run.sh passes
  • Commit: memory: brain server registered, rewire outcomes

[Claude Code, brain-latency session]