Skip to content

Semantic Search, /recall

What it indexes

  • memory/ (daily logs, general, handoffs, sessions)
  • nexus/ (shared brain: fleet, architecture, steering)
  • docs/ (infra docs, playbooks, operations)
  • brands/ (brand context)
  • CLAUDE.md (bootstrap)

Skips binary/large files (>1MB), .venv, node_modules, .git, screenshots.

Stack

  • Embeddings: fastembed with BAAI/bge-small-en-v1.5 (ONNX, 384-dim, ~150MB footprint). CPU-only.
  • Storage: sqlite + sqlite-vec extension. DB at data/search/index.db.
  • Chunking: Markdown-aware, splits by heading then by 800-char windows with 100-char overlap.
  • Venv: ~/.forge-venvs/search/ (independent from assets/ and integrations/ venvs).

v2 three-layer retrieval (2026-05-04)

/recall no longer dumps full chunk bodies on every query. Three modes:

Layer Flag Returns
1, search bare query [chunk_id] path date \| summary score=0.NN per hit
2, timeline --timeline <id> --window N Same-file neighbors ordered by chunk_idx, anchor marked
3, details --get a3f2,b8c1 Full chunk bodies for explicit IDs

Layer 1 is gated behind FORGE_RECALL_V2=1 during the dogfood week; without the env var, the bare query form still prints v1-style full bodies. Use --full to force v1 behavior even with the env var set.

Schema additions backing v2: - chunks.chunk_id (4-hex sha1 of path:chunk_idx, collision-extended) - chunks.summary (first 80 chars, word-boundary truncation, ellipsized) - meta(key, value) table tracking schema_version=2

v1->v2 upgrade drops chunks + chunk_vecs and clears files to force a single full re-embed (~1-2 min on current corpus, ~5200 chunks).

Invocation

Via skill (preferred in-session): 

/recall how does the task dispatcher route workers
/recall -k 10 eight sleep credentials
/recall --json frigate config
/recall --get 1b14
/recall --timeline 1b14 --window 3
/recall --full <query>

Direct CLI: 

FORGE_RECALL_V2=1 /home/justinwieb/.forge-venvs/search/bin/python \
  /home/justinwieb/forge/scripts/forge_search_query.py "query here"

Reindex (incremental, only changed files): 

/home/justinwieb/.forge-venvs/search/bin/python \
  /home/justinwieb/forge/scripts/forge_search_index.py

Full rebuild: 

/home/justinwieb/.forge-venvs/search/bin/python \
  /home/justinwieb/forge/scripts/forge_search_index.py --rebuild

Maintenance

  • Nightly cron at 03:30 re-runs forge_search_index.py --quiet to logs/search-index.log.
  • First run after corpus change: ~1-2 min for ~5200 chunks on Console CPU.
  • Incremental runs complete in seconds.
  • Bench harness scripts/forge_recall_bench.py logs v1-vs-v2 sizes to data/recall-bench/<date>.jsonl.

When to use vs grep

  • grep for exact strings, literal filenames, or known symbols.
  • /recall for "where did we decide", "how was X set up", fuzzy concepts.
  • /recall --timeline once you've found the right chunk and want surrounding context.
  • /recall --get once you've found the right chunk and want full body.

Files

  • scripts/forge_search_index.py, walker + chunker + embedder
  • scripts/forge_search_query.py, three-layer CLI frontend
  • scripts/forge_recall_bench.py, v1-vs-v2 bench harness
  • .claude/skills/recall/SKILL.md, slash-command wrapper
  • data/search/index.db, sqlite DB (gitignored, ~3-20MB depending on corpus)
  • data/search/models/, cached ONNX model
  • data/recall-bench/<date>.jsonl, bench results