URL: https://mkdocs.justinsforge.com/memory/general/reference_forge_backup_strategy/

Forge Backup Strategy¶

Forge lives at /home/justinwieb/forge on Console (VM 103, 192.168.86.50), a 24 GB root disk that is currently 89% full (20 GB used, 2.6 GB free). The repo is ~266 MB on disk. Git remote: https://github.com/JustinWieb/forge.git. This document covers what needs backing up, where each piece goes, and concrete cron entries with retention windows.

1. What Is Already Covered¶

Git Remote (GitHub)¶

git remote -v confirms origin https://github.com/JustinWieb/forge.git. Everything committed to the repo and pushed is durable. This covers:

scripts/ (all Python, bash, and helper scripts)
brands/ (READMEs, Shopify locales, brand config)
infra/ (systemd units, logrotate config, n8n scaffolds, context-api source)
system-map/ (architecture.md, fleet.md, steering.md, google-drive.md)
CLAUDE.md, FORGE-DOCTRINE.md, LESSONS.md, README.md
memory/ (handoffs, daily logs, general topic files that are checked in)
data/time_categories.json, data/README.md, and other small committed data files

Gap: Anything with unstaged or uncommitted changes is NOT protected. The current git status shows numerous modified files (locales, READMEs, LESSONS.md) and deleted files staged but not committed. A commit discipline gap means the remote lags the working tree.

Finn Workspace Nightly Rsync¶

A nightly rsync at 02:00 on Finn copies /mnt/pve/fast-storage/ (the 8TB NVMe) to /mnt/storage/workspace-backup/ with versioned deltas at /mnt/storage/workspace-backup-versions/YYYY-MM-DD/. However, Forge itself is NOT on the NVMe; it lives on Console's local disk (/home/justinwieb/), which is an Ubuntu VM disk image on Finn but not in the /mnt/pve/fast-storage/ share. Console's root disk is NOT caught by this rsync.

2. What Is NOT Currently Backed Up¶

Asset	Location	Risk
`forge/` working tree (uncommitted changes)	Console `/home/justinwieb/forge/`	Lost if Console VM disk corrupts
`~/.claude/` (project memory, skills, settings)	Console `/home/justinwieb/.claude/`	All session memory, feedback files, reference files
`~/.forge-secrets/`	Console `/home/justinwieb/.forge-secrets/`	Credentials (DO NOT back up to git or Drive; covered separately below)
`~/.config/rclone/rclone.conf`	Console `/home/justinwieb/.config/`	Drive auth; losing it requires OAuth re-auth
`forge/data/` (runtime JSONL, context DB, gdrive-cache)	`/home/justinwieb/forge/data/`	Hevy, quota, telegram chat history, search DB
`forge/logs/`	`/home/justinwieb/forge/logs/`	Operational logs (lower priority, but useful for incident review)
n8n SQLite DB	CT 106 `/opt/n8n/` (inside container)	All n8n workflows and credentials
n8n Docker config	CT 106 `/opt/n8n/docker-compose.yml`	Workflow engine config

3. Backup Destinations: What Goes Where¶

Rule: Never Put Secrets in Git or Drive¶

~/.forge-secrets/ stays out of all backup paths that touch git, GitHub, or shared drives. The correct backup for secrets is an encrypted local copy to /mnt/storage/forge-secrets-encrypted/ using gpg --symmetric, plus manual export to NordPass for critical credentials.

Google Drive (via rclone `gdrive:`)¶

Use for: human-readable assets and docs that Justin might need to recover from any device.

What	Source	Drive Destination
Forge `memory/` snapshot	`/home/justinwieb/forge/memory/`	`gdrive:Forge-Backups/memory/YYYY-MM-DD/`
Claude memory layer	`/home/justinwieb/.claude/projects/-home-justinwieb-forge/memory/`	`gdrive:Forge-Backups/claude-memory/YYYY-MM-DD/`
Forge critical configs	`CLAUDE.md`, `FORGE-DOCTRINE.md`, `LESSONS.md`	`gdrive:Forge-Backups/root-docs/YYYY-MM-DD/`

Drive retention: keep last 30 daily snapshots, then one per week for 90 days, then one per month indefinitely. Implement with a sweep that deletes daily snapshots older than 30 days and weekly ones older than 90 days.

Finn Media Drive (`/mnt/storage/` via NFS from Console)¶

Use for: full working-tree tarballs and n8n database. Fast LAN transfer, large capacity.

What	Source	Finn Destination
Full forge tarball	`/home/justinwieb/forge/`	`/mnt/storage/forge-backups/daily/forge-YYYY-MM-DD.tar.gz`
Claude home dir	`/home/justinwieb/.claude/`	`/mnt/storage/forge-backups/daily/claude-YYYY-MM-DD.tar.gz`
n8n DB + config	CT 106 `/opt/n8n/` (rsync via SSH)	`/mnt/storage/forge-backups/n8n/n8n-YYYY-MM-DD.tar.gz`
rclone config	`/home/justinwieb/.config/rclone/`	`/mnt/storage/forge-backups/config/rclone-YYYY-MM-DD.tar.gz`
Secrets (encrypted)	`/home/justinwieb/.forge-secrets/`	`/mnt/storage/forge-backups/secrets-encrypted/secrets-YYYY-MM-DD.tar.gz.gpg`

Finn retention: keep last 14 daily tarballs, 8 weekly tarballs, 12 monthly tarballs. Implemented by the proposed forge_backup.sh script below.

Both (Drive AND Finn)¶

The Claude memory layer (~/.claude/projects/-home-justinwieb-forge/memory/) is small (596 KB) and high-value (all learned preferences, feedback corrections, reference files). Write it to both destinations nightly.

4. Script: `forge_backup.sh` (SHIPPED 2026-04-30)¶

Location: /home/justinwieb/forge/scripts/forge_backup.sh

Uses rclone directly for Drive (not forge_gdrive_write.py) for directory-level efficiency. Cron wired at 06:30 UTC. GPG passphrase created at ~/.forge-secrets/.backup-passphrase (passphrase must be saved to NordPass before relying on secrets backup). Smoke tested 2026-04-30: forge tarball 124 MB to Finn, Drive memory sync 130 files OK.

n8n volume path confirmed: /var/lib/docker/volumes/n8n_n8n_data/_data on host n8n.

#!/usr/bin/env bash
# forge_backup.sh — nightly forge backup to Finn media drive + Google Drive
# Cron: 01:30 CT daily (06:30 UTC)
# Log: /home/justinwieb/forge/logs/backup.log
set -euo pipefail

FORGE_DIR="/home/justinwieb/forge"
CLAUDE_MEM="/home/justinwieb/.claude/projects/-home-justinwieb-forge/memory"
RCLONE_CONF="/home/justinwieb/.config/rclone"
SECRETS_DIR="/home/justinwieb/.forge-secrets"
BACKUP_BASE="/mnt/storage/forge-backups"
GDRIVE_BASE="Forge-Backups"
DATE=$(date +%Y-%m-%d)
LOG="$FORGE_DIR/logs/backup.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; }

log "=== forge backup start $DATE ==="

# --- Finn: full forge tarball ---
mkdir -p "$BACKUP_BASE/daily"
FORGE_TAR="$BACKUP_BASE/daily/forge-$DATE.tar.gz"
tar --exclude="$FORGE_DIR/logs" \
    --exclude="$FORGE_DIR/data/search" \
    --exclude="$FORGE_DIR/data/gdrive-cache" \
    --exclude="$FORGE_DIR/scripts/__pycache__" \
    -czf "$FORGE_TAR" -C /home/justinwieb forge
log "forge tarball: $FORGE_TAR ($(du -sh "$FORGE_TAR" | cut -f1))"

# --- Finn: claude memory tarball ---
CLAUDE_TAR="$BACKUP_BASE/daily/claude-$DATE.tar.gz"
tar -czf "$CLAUDE_TAR" -C /home/justinwieb .claude
log "claude tarball: $CLAUDE_TAR"

# --- Finn: rclone config ---
mkdir -p "$BACKUP_BASE/config"
RCLONE_TAR="$BACKUP_BASE/config/rclone-$DATE.tar.gz"
tar -czf "$RCLONE_TAR" -C /home/justinwieb ".config/rclone"
log "rclone config tarball: $RCLONE_TAR"

# --- Finn: n8n DB (stop container, copy, restart) ---
mkdir -p "$BACKUP_BASE/n8n"
N8N_TAR="$BACKUP_BASE/n8n/n8n-$DATE.tar.gz"
ssh n8n "docker compose -f /opt/n8n/docker-compose.yml stop && \
    tar -czf /tmp/n8n-backup.tar.gz -C /opt/n8n . && \
    docker compose -f /opt/n8n/docker-compose.yml start" && \
    scp n8n:/tmp/n8n-backup.tar.gz "$N8N_TAR" && \
    ssh n8n "rm /tmp/n8n-backup.tar.gz"
log "n8n tarball: $N8N_TAR"

# --- Finn: secrets encrypted ---
mkdir -p "$BACKUP_BASE/secrets-encrypted"
SECRETS_TAR="$BACKUP_BASE/secrets-encrypted/secrets-$DATE.tar.gz.gpg"
tar -cz -C /home/justinwieb .forge-secrets | \
    gpg --batch --yes --passphrase-file "$SECRETS_DIR/.backup-passphrase" \
        --symmetric --output "$SECRETS_TAR"
log "secrets encrypted: $SECRETS_TAR"

# --- Drive: memory snapshot (uses forge_gdrive_write.py) ---
GDRIVE_MEM_DEST="$GDRIVE_BASE/memory/$DATE"
GDRIVE_WRITE="$FORGE_DIR/scripts/forge_gdrive_write.py"
for f in "$FORGE_DIR/memory/general/"*.md; do
    fname=$(basename "$f")
    python3 "$GDRIVE_WRITE" "$f" "$GDRIVE_MEM_DEST/$fname" --mkdir-parents 2>>"$LOG" || true
done
log "Drive memory snapshot: $GDRIVE_MEM_DEST"

# --- Drive: claude memory snapshot ---
GDRIVE_CLAUDE_DEST="$GDRIVE_BASE/claude-memory/$DATE"
for f in "$CLAUDE_MEM/"*.md; do
    fname=$(basename "$f")
    python3 "$GDRIVE_WRITE" "$f" "$GDRIVE_CLAUDE_DEST/$fname" --mkdir-parents 2>>"$LOG" || true
done
log "Drive claude-memory snapshot: $GDRIVE_CLAUDE_DEST"

# --- Drive: root docs ---
GDRIVE_DOCS_DEST="$GDRIVE_BASE/root-docs/$DATE"
for doc in CLAUDE.md FORGE-DOCTRINE.md LESSONS.md README.md; do
    python3 "$GDRIVE_WRITE" "$FORGE_DIR/$doc" "$GDRIVE_DOCS_DEST/$doc" --mkdir-parents 2>>"$LOG" || true
done
log "Drive root docs: $GDRIVE_DOCS_DEST"

# --- Finn retention sweep ---
# Keep 14 daily, 8 weekly (Sundays), 12 monthly (1st of month)
for dir in daily n8n; do
    find "$BACKUP_BASE/$dir" -name "*.tar.gz" -mtime +14 | while read -r f; do
        bname=$(basename "$f")
        fdate=$(echo "$bname" | grep -oP '\d{4}-\d{2}-\d{2}')
        dow=$(date -d "$fdate" +%u 2>/dev/null || echo "0")
        dom=$(date -d "$fdate" +%d 2>/dev/null || echo "0")
        age=$(( ( $(date +%s) - $(date -d "$fdate" +%s 2>/dev/null || echo 0) ) / 86400 ))
        # Keep Sundays for 8 weeks (56 days)
        if [[ "$dow" == "7" && "$age" -lt 57 ]]; then continue; fi
        # Keep 1st of month for 12 months (365 days)
        if [[ "$dom" == "01" && "$age" -lt 366 ]]; then continue; fi
        log "pruning $f (age ${age}d)"
        rm -f "$f"
    done
done

log "=== forge backup done ==="

5. Cron Entries¶

Add these to the justinwieb crontab. Uses UTC. 06:30 UTC = 01:30 CT (CDT), well after auto-dream (04:00 UTC) and quota aggregator (04:30 UTC) finish.

# forge-backup begin
# Full forge backup: Finn media drive tarballs + Drive memory snapshots. 01:30 CT = 06:30 UTC.
30 6 * * * /home/justinwieb/forge/scripts/forge_backup.sh >> /home/justinwieb/forge/logs/backup.log 2>&1
# forge-backup end

No separate weekly or monthly cron is needed; the retention sweep inside forge_backup.sh handles the promotion logic.

6. Nightly Sequence (Consolidated View)¶

After adding the backup job the nightly schedule on Console looks like this:

UTC	CT (CDT)	Job
03:00	22:00	Eval harness (`forge_eval_harness.py`)
03:30	22:30	Search index rebuild (`forge_search_index.py`)
04:00	23:00	Auto-dream memory consolidation (`forge_memory_auto_dream.py`)
04:15	23:15	Retention sweep (`forge_cleanup_retention.sh`)
04:30	23:30	Quota aggregator (`forge_quota_tracker.py`)
06:30	01:30	Forge backup (`forge_backup.sh`) -- new

The backup runs last, after all mutation jobs, so the snapshot captures a settled state.

7. n8n Backup Note¶

The n8n SQLite database lives at CT 106 (forge-n8n, 192.168.86.82). Per reference_n8n_state.md: never docker cp the DB while the container runs (WAL corruption risk). The proposed forge_backup.sh stops the container, tars /opt/n8n/, restarts, then copies the tar over SSH. This produces a consistent snapshot at the cost of ~30 seconds of n8n downtime at 01:30 CT, which is acceptable. If that window becomes a problem, the alternative is sqlite3 /opt/n8n/data/database.sqlite ".backup /tmp/n8n.db" (WAL-safe online backup) before stopping.

8. Secrets Backup¶

The .backup-passphrase file referenced above must be created manually:

# one-time setup on Console
openssl rand -base64 32 > ~/.forge-secrets/.backup-passphrase
chmod 600 ~/.forge-secrets/.backup-passphrase
# write the passphrase into NordPass manually before relying on this

The encrypted tarball at /mnt/storage/forge-backups/secrets-encrypted/ is useless without the passphrase, which must be stored in NordPass (the canonical credential store per doctrine Section security rules).

9. Console VM Disk (Worst-Case Scenario)¶

All of the above backs up the Forge working tree, but the Console VM disk image itself (CT/VM 103 on Finn) is not snapshotted via Proxmox Backup Server (PBS) by default. Adding a PBS backup job for VM 103 at the Proxmox level gives a full VM restore point independent of any file-level backup. This is out of scope for this document but is the recommended next step for full disaster recovery coverage.

10. What to Skip¶

Do not back up: - forge/logs/ (large, ephemeral, logrotate already handles rotation) - forge/data/search/ (sqlite-vec index, fully rebuilt by forge_search_index.py from source files) - forge/data/gdrive-cache/ (fully rebuilt by forge_gdrive_index_extend.py) - forge/scripts/__pycache__/ (bytecode, auto-regenerated) - forge/assets/asset-drop/ (raw binary drops, not committed, treated as ephemeral staging)

[Claude Code, 2026-04-30]