Skip to content

Plan: Self-hosted editor-native clip library (MAM) for Gus The Bass

URL: https://mkdocs.justinsforge.com/memory/plans/gus-clip-library-mam-2026-06-15/

Date: 2026-06-15

Approved spec: A self-hosted, editor-native clip library (generalist, brand-tagged, Gus The Bass first ingest) modeled on Immich's speed and navigation. Ingest footage to the NVMe and pre-generate 720p proxies + thumbnails. Catalog lives in the forge-data lifeos Postgres spine (CT 109) as a workspace-scoped clips domain (revised 2026-06-15 from standalone SQLite, per single-source-of-truth: forge-data already runs Postgres 17 + pgvector + multi-tenant RLS and seeds a gus-the-bass workspace). Columns: hard metadata (capture date, GPS, duration, resolution, brand) + Michael's Apple captions/albums (identity) + AI action/scene tags + CLIP embedding (vector, pgvector). Proxies + thumbnails stay as files on the NVMe (never in the DB). Front-end clips.justinsforge.com: justified virtualized grid, instant proxy playback, hover-scrub, search with year/fish/action filters. Selection feeds a pluggable Actions layer; first action = Send to Premiere (FCP7 .xml bin into a project's Project/ folder, validated 2026-06-15). Plus collections and custom poster-frame. Behind Cloudflare Access. Proxies double as Premiere edit proxies.

Out of scope (this round): - Premiere UXP panel for in-app drag-to-timeline (Phase 2; web import-to-bin is the handoff for now). - Actions beyond Send-to-Premiere + download (iPhone push, email-link, BLIP/other-tool enrichment come after core works). - In/out sub-clip trimming before send (send whole clips first). - Review/approval/commenting (Frame.io-style); not needed yet. - Full icloudpd migration is its own track, gated on Michael's Apple ID app-specific password. The catalog is built and proven against footage already on /mnt/workspace first, so the build never blocks on the cloud pull.

Prerequisites / decisions baked in: - Catalog DB = forge-data lifeos Postgres (CT 109, 192.168.86.86), NOT SQLite. Clip tables added as migration 0008_clips.sql (next free number; 0001-0007 applied) under data/workspace/migrations/, in their own media schema (same pattern as the existing tasks/habits/docs/inbox domains), workspace-scoped via core.apply_workspace_rls('media.<table>'), into the seeded gus-the-bass workspace. All access through the existing forge_workspace_db.py helper (app role + set_identity/app_request, RLS-bound). Migrations via forge_workspace_migrate.py. Backup rides the existing nightly forge_workspace_backup.timer (no separate backup). Verified ready 2026-06-16: pgvector 0.8.0, workspace seeded, RLS helpers present, Console can connect. - Install gaps to clear before their tasks: torch + open_clip are NOT installed and Console CPU lacks AVX2, so the CLIP embed pass (Task 4) runs on Vector's RTX 5070 via /sshrequest (--device cuda), not Console. icloudpd is NOT installed and ~/.forge-secrets/icloud-michael.env does not exist yet (both needed only for Task 10). - Vector search: CLIP embedding stored in a pgvector vector column with an HNSW/IVFFlat index; query with the <=> cosine operator. (pgvector is installed + trusted on CT 109.) - CLIP model: open_clip ViT-B-32 (laion2b) for shared image+text embeddings. Bulk embed pass can run on Vector's GPU via /sshrequest; incremental runs on Console CPU. - Heavy media artifacts (proxies, thumbs) live as files on the NVMe at /mnt/workspace/Forge/clip-library/{proxies,thumbs} (never in Postgres); the DB stores their paths. - Reuse existing helpers: forge_workspace_db.py (DB), forge_gmail.py (email), Telegram notify-out bot (phone push), Cloudflare tunnel + /cf-add (ingress), Cloudflare Access (auth). Never hand-roll auth. - icloudpd creds: ~/.forge-secrets/icloud-michael.env (chmod 600). DB creds: existing ~/.forge-secrets/forge-data.env.

Task list

Task 1: Generalize the FCP7 XML builder from the CES test

  • Files: create /home/justinwieb/forge/scripts/forge_premiere_xml_builder.py
  • What: Lift today's validated CES .xml into a reusable module/CLI. Input: list of clip paths (+ per-clip probed duration/fps/res via ffprobe), output mode (bin or sequence), target resolution(s), path-style (windows W: or posix), bin/sequence name. Emits FCP7 xmeml .xml. Probe data gathered with ffprobe; integer-frame durations.
  • Verification: python3 /home/justinwieb/forge/scripts/forge_premiere_xml_builder.py --clips "/mnt/workspace/JustinWieb-VR/2026/2026-01-04_CES-Day-1-Unveil/Assets/IMG_2691.MOV" --mode bin --path-style windows --out /tmp/test_bin.xml && python3 -c "import xml.dom.minidom; xml.dom.minidom.parse('/tmp/test_bin.xml'); print('OK')"
  • Commit: clip-library: generalize CES FCP7 xml builder into forge_premiere_xml_builder

Task 2: Catalog schema migration (Postgres spine) + DB helper

  • Files: create /home/justinwieb/forge/data/workspace/migrations/0008_clips.sql; create /home/justinwieb/forge/scripts/forge_clip_catalog.py
  • What: Postgres migration following the 0004_tasks.sql recipe: CREATE SCHEMA media; tables media.clips (id, sha1 unique, path, workspace_id, brand, capture_ts, gps_lat, gps_lon, duration_s, width, height, fps, codec, apple_caption, favorite, poster_frame_path, proxy_path, thumb_path, ai_tags jsonb, description, embedding vector(512), ingested_ts), media.collections, media.clip_collections, plus a tsvector/GIN index over caption+tags+description and an HNSW index on embedding; GRANT USAGE/DML on schema media to lifeos_app + ALTER DEFAULT PRIVILEGES; then SELECT core.apply_workspace_rls('media.<table>') per table. forge_clip_catalog.py wraps forge_workspace_db.py for idempotent upsert keyed on sha1 (re-ingest safe), run under the app role with identity set.
  • Verification: python3 /home/justinwieb/forge/scripts/forge_workspace_migrate.py && python3 -c "import sys;sys.path.insert(0,'/home/justinwieb/forge/scripts');import forge_clip_catalog as c;print(c.smoke_test())" then python3 /home/justinwieb/forge/scripts/forge_workspace_rls_test.py (RLS gate still passes with the new tables).
  • Commit: clip-library: clips domain migration on lifeos Postgres + RLS + catalog helper

Task 3: Metadata + proxy + thumbnail ingest

  • Files: create /home/justinwieb/forge/scripts/forge_clip_ingest.py; output dirs /mnt/workspace/Forge/clip-library/{proxies,thumbs}
  • What: Walk a path, for each video: sha1, ffprobe (duration/fps/res/codec/capture_ts), exiftool (Apple caption, keywords, Favorite, GPS), generate 720p h264 proxy (-vf scale=-2:720) and a mid-clip thumbnail (ffmpeg -nostdin), upsert into catalog. Idempotent (skip if sha1 present + proxy exists). Brand inferred from top-level workspace dir.
  • Verification: python3 /home/justinwieb/forge/scripts/forge_clip_ingest.py --path "/mnt/workspace/JustinWieb-VR/2026/2026-01-04_CES-Day-1-Unveil/Assets" --limit 5 && python3 -c "import sys;sys.path.insert(0,'/home/justinwieb/forge/scripts');import forge_clip_catalog as c;print(c.count(), 'clips; proxies:', __import__('glob').glob('/mnt/workspace/Forge/clip-library/proxies/*')[:2])"
  • Commit: clip-library: metadata + proxy + thumbnail ingest pipeline

Task 4: CLIP embeddings + AI action/scene tags

  • Files: create /home/justinwieb/forge/scripts/forge_clip_ai_tags.py
  • What: For each catalog clip lacking an embedding: sample keyframes (ffmpeg scene-detect, cap N), run open_clip ViT-B-32 to produce an averaged image embedding (written to the pgvector embedding column) + coarse zero-shot tags against a fishkeeping/outdoor label set (eating, feeding, swimming, tank, underwater, boat, etc.). Whisper transcript only if an audio-energy/voice check passes (most clips silent). --device cpu|cuda.
  • Verification: python3 /home/justinwieb/forge/scripts/forge_clip_ai_tags.py --limit 5 --device cpu && python3 -c "import sys;sys.path.insert(0,'/home/justinwieb/forge/scripts');import forge_clip_catalog as c;print(c.embedding_coverage())"
  • Commit: clip-library: CLIP embeddings + zero-shot action/scene tagging

Task 5: Search API (FastAPI on Console)

  • Files: create /home/justinwieb/forge/scripts/forge_clip_search_api.py; systemd unit /home/justinwieb/.config/systemd/user/forge-clip-search-api.service
  • What: FastAPI on 127.0.0.1:7360, all DB access via forge_workspace_db.py app role with per-request set_identity (RLS-bound to the caller's workspace). Endpoints: GET /search?q=&year=&brand=&fish=&action=&limit= (CLIP text-embedding cosine via pgvector <=> + tsvector text match + filters, ranked in SQL), GET /clip/{id}, GET /proxy/{id} and GET /thumb/{id} (stream files from NVMe), GET /collections, POST /collections, POST /clip/{id}/poster (set poster frame), POST /send-to-premiere (clip ids + mode + target project dir -> calls Task 1 builder, writes .xml).
  • Verification: systemctl --user start forge-clip-search-api && sleep 2 && curl -fsS "http://127.0.0.1:7360/search?q=fish&limit=3" | python3 -m json.tool | head
  • Commit: clip-library: FastAPI search + send-to-premiere service

Task 6: Send-to-Premiere end-to-end test

  • Files: none new (integration test); optional /home/justinwieb/forge/scripts/tests/test_clip_send_to_premiere.sh
  • What: Hit /send-to-premiere with 3 real Gus clip ids, mode bin, path-style windows, into a scratch project's Project/ dir; confirm a well-formed .xml with W: paths is written.
  • Verification: bash /home/justinwieb/forge/scripts/tests/test_clip_send_to_premiere.sh (asserts xml well-formed + 3 file refs). Manual confirm: Justin imports it in Premiere once.
  • Commit: clip-library: send-to-premiere integration test
  • Files: create /home/justinwieb/forge/sites/clip-library/ (SPA: index.html, app.js, styles); served by the API or nginx
  • What: Use the frontend-design skill. Search bar + Year/Fish/Action filter chips; justified virtualized thumbnail grid with infinite scroll; click opens proxy player (instant playback, hover-scrub); multi-select checkboxes; selection tray with target-project dropdown + "Send to Premiere"; collections sidebar; set-poster-frame control. Talks to the Task 5 API. Home pill per feedback_home_button_on_subdomains.
  • Verification: /preview-site clip-library then load the Tailscale URL; search returns a grid, a proxy plays, selecting 2 clips + Send writes an .xml.
  • Commit: clip-library: Immich-style web search + select UI

Task 8: Publish clips.justinsforge.com behind Cloudflare Access

  • Files: Cloudflare tunnel ingress (via /cf-add clips justinsforge.com http://127.0.0.1:7360); add Access policy; add home-landing tile per feedback_new_subdomain_links_home
  • What: Route clips.justinsforge.com to the local API/site, protect with Cloudflare Access (Gus business footage). Add a tile on the justinsforge home landing.
  • Verification: curl -fsS -o /dev/null -w "%{http_code}" https://clips.justinsforge.com returns 302/200 to Access; authenticated load shows the grid.
  • Commit: clip-library: publish clips.justinsforge.com behind Cloudflare Access

Task 9: Collections + custom poster-frame UX

  • Files: edits to forge_clip_search_api.py, sites/clip-library/app.js
  • What: Finalize named collections (create, add/remove clips, open a collection as a filtered view) and poster-frame selection (scrub to a frame, "set as thumbnail", regenerates thumb). These are the editor features Immich lacks.
  • Verification: Create a collection via UI, add 3 clips, reopen it (3 shown); set a poster frame, thumb updates in grid.
  • Commit: clip-library: collections + custom poster-frame

Task 10: icloudpd migration track (gated on Michael's Apple ID)

  • Files: create /home/justinwieb/forge/scripts/forge_icloud_video_pull.py; secret ~/.forge-secrets/icloud-michael.env
  • What: Wrap icloudpd: download originals (video focus) to /mnt/workspace/Gus-The-Bass/source/YYYY/, resumable, write a manifest (filename, size, sha1) and a verify step that diffs against the iCloud asset count before any cloud deletion is ever considered. Never deletes anything. Then run Task 3 + Task 4 ingest over the pulled tree.
  • Verification: dry-run lists asset count without downloading: python3 /home/justinwieb/forge/scripts/forge_icloud_video_pull.py --dry-run prints total assets + planned target paths. (Full pull run separately once creds are in place.)
  • Commit: clip-library: resumable icloudpd video migration with manifest + verify

Task 11: Register and document

  • Files: memory/general/reference_clip_library_mam.md (with mkdocs URL line); update MEMORY.md index; smoke test
  • What: Topic file documenting the stack (scripts, DB schema, API endpoints, the FCP7 .xml handoff, Immich-as-UX-reference, build order). Index line in MEMORY.md. Per feedback_register_tools.md.
  • Verification: bash /home/justinwieb/forge/scripts/forge_eval_run.sh
  • Commit: clip-library: register + document the MAM stack