Reboot Resilience Stack

Built ahead of the Finn BIOS power-on policy drill, validated end-to-end on 2026-04-30 (stage 4 full host reboot). Goal: any reboot of Console or Finn returns every service automatically and the active Claude conversation comes back with full history.

The five pieces

Piece	Path	Trigger	What it does
Fleet probe	forge/scripts/forge_fleet_probe.sh	manual + boot	ICMP all 7 production hosts/CTs + HTTPS check on mkdocs/dashboard/usage. Stdout summary, optional --json. Exit code = fail count.
Pre-reboot save	forge/scripts/forge_pre_reboot_save.sh	manual, run BEFORE any planned reboot	Walks ~/.claude/sessions/*.json for sessions touched in the last 4h with a live JSONL, writes UUIDs to ~/.forge-last-session (TSV: uuid, bridgeId, name, age).
Boot notify	forge/scripts/forge_boot_notify.sh + forge-boot-notify.service	systemd at boot	Reports Console kernel/uptime + Finn uptime (via SSH) + fleet probe to @forge_notify_outbound_bot. Writes marker to forge/notifications/boot-markers/.
Post-boot sessions	forge/scripts/forge_post_boot_sessions.sh + forge-post-boot-sessions.service	systemd at boot, after boot-notify	Spawns 2 Opus 4.7 sessions on the homebase tmux socket: post-boot-recap_Opus47 (resumed from saved UUID, primed with boot marker + probe + daily log + handoffs) and post-boot-blank_Opus47 (fresh, ready). Each session auto-runs /remote-control so it shows up in claude.ai/code.
/resume skill	forge/.claude/skills/resume/SKILL.md	user-invoked any time	Resurrect any old chat. /resume lists last 15. /resume <uuid>, /resume <claude.ai/code URL>, or /resume <fuzzy text> all work.

The flow

Planned reboot

1. bash forge/scripts/forge_pre_reboot_save.sh    # snapshot active UUIDs
2. ssh finn 'reboot'                              # or pct reboot, or shutdown
3. (everything goes dark for 3-5 min)
4. Console boots → forge-boot-notify fires → Telegram pings you
5. forge-post-boot-sessions fires → recap session resumes the saved UUID
6. Reconnect from any device → claude.ai/code → finn-reboot session is there

Ad-hoc / unplanned reboot (no pre-save)

1. (power cuts, kernel panic, whatever)
2. Console boots → forge-boot-notify fires
3. forge-post-boot-sessions checks ~/.forge-last-session:
   - If <6h old: resume that UUID
   - If >6h old or missing: fall through to most-recently-modified JSONL
4. /resume skill always available to resurrect anything specific by hand

The 6h staleness guard prevents a week-old saved file from auto-resurrecting a long-dead conversation on a random reboot.

Verified recovery times (stage 4 dress rehearsal)

Marker	Elapsed
reboot issued → Finn host back	~4 min
→ Console boot-notify fired	~6.5 min
→ fleet probe 9/10 (n8n still warming)	~6.5 min
→ fleet probe 10/10	~16 min

n8n (CT 106) lags because it's order=5 alongside the Console VM (order=5). Could bump to order=4, or add a retry loop to boot-notify so the summary reflects steady state. Not blocking.

CT startup ordering on Finn

Tier	order=	CTs
1	1	adguard (DNS first)
2	3	plex, media-server
3	4	immich, frigate
4	5	n8n, Console VM 103

up=15 (or 30 for Console) on each. onboot=1 everywhere except minecraft (110, on demand).

Drill stages run

• ✅ 1: Console reboot (today, post-AVX passthrough)
• ✅ 2: pct reboot 108 frigate, recovered in 21s
• ✅ 3: pct reboot 105 adguard, recovered in 4s
• ✅ 4: ssh finn reboot full host, full green in 16 min
• ⏳ 5: BIOS trip (set "State After G3"→S0 + configure BMC LAN, suggest 192.168.86.68)
• ⏳ 6: Pull Tapo plug, validate BIOS policy

Each stage uses forge_fleet_probe.sh as pass/fail gate.

Pending (not blocking the drill)

• BMC LAN configure, bundles with stage 5
• UPS + NUT (forensic-grade truth on next power event)
• Encrypted secrets backup to Drive
• forge-tmux-anchor Type=forking + tmux-resurrect (B7 from finn-recovery handoff). Briefing written 2026-04-30 at forge/memory/handoffs/spawn-tmux-harness-2026-04-30.md after a /tmp/tmux-1000/default stale-socket bricked remote-bridge /spawn. Harness pre-flights sockets, isolates per-session (-L spawn-<name>), gates post-boot-sessions on a sweep oneshot. NOT YET SHIPPED. Immediate unblock = rm /tmp/tmux-1000/default on Console before any reboot.

[Claude Code]