Skip to content

Clip Library: Findings & Decisions (handoff for the workspace bot)

URL: https://mkdocs.justinsforge.com/memory/handoffs/clip-library-findings-2026-06-17/

Date: 2026-06-17. Author: [Claude Code], with Justin. Purpose: brief the workspace-suite bot on the clip-library module so it builds it correctly on top of the suite foundation. Read alongside: - Suite architecture: memory/plans/workspace-suite-architecture-and-rebuild-plan-2026-06-17.md - Clip-library task plan: memory/plans/gus-clip-library-mam-2026-06-15.md - Caption ingest detail: memory/general/reference_apple_photos_caption_ingest.md

1. Goal

A private, self-hosted, searchable video library for Michael's footage (Gus The Bass first, generalist for all brands later). Own the data on the NVMe, serve it from clips.justinsforge.com, make 5-10k clips searchable by typing ("Gus eating minnows"), and feed selected clips straight into Premiere to make long-form editing fast.

2. Constraints (hard)

  • Michael has 5-10k videos on iCloud (shown on his phone, actually in iCloud).
  • Michael has NO Mac. Only an iPhone + a Windows PC. (Rules out any Mac-only tool.)
  • Want all clips copied to the server (NVMe) so we own the data and serve it directly; nothing trapped in iCloud.
  • Human captions like "Gus eating minnows" are high value, but captioning must be ONE easy step or it will not get done.
  • Future: Michael keeps shooting on iPhone; he'll copy new clips to the server (Files app or external drive). Still wants to type quick notes from the iPhone.

3. Key finding (a correction)

The caption typed in Apple Photos (the Description field) and the Favorite flag are stored in the Photos library database, NOT embedded in the .mov file. - icloudpd downloads the file bytes + basic EXIF (date, sometimes GPS) but does NOT bring the typed caption or favorite. - osxphotos could export them, but it is Mac-only -> ruled out (no Mac). - Michael has only typed ~50 captions so far, so there is no large body of Apple captions worth salvaging. (A fragile iCloud-web-API pull was considered and is now unnecessary; he can just re-add those 50 in our app.)

4. Decided approach: decouple bytes from captions

  1. Bytes: icloudpd pulls all 5-10k videos from iCloud to the NVMe. Needs only Michael's Apple ID app-specific password, no Mac, runs headless. Resumable; manifest + verify; never deletes from iCloud.
  2. Captions/favorites: captured in our own app (clips.justinsforge.com as an iPhone PWA) in a fast review mode designed to match the Apple Photos swipe-down speed: big preview, tap heart = favorite, type one line, swipe to next. Data lands directly in our database. ONE surface for the old library and all future clips. HARD requirement: thumb-speed, one easy step, or it won't be used.
  3. Future capture (new shoots): copy the clip to the server, then caption in our app; or, later, an iOS Shortcut that uploads selected clips + a typed note in one share-sheet tap.
  4. AI covers the bulk: content recognition (CLIP), any spoken words (Whisper only where speech exists; most clips are silent), plus dates/metadata make the rest findable. Human captions are only needed for favorites/hero clips, so the manual burden is bounded.

5. How this fits the workspace suite (important)

The clip library is a module of the personal-OS suite, not a separate app. It is the Immich-shaped module (the suite's own template): a custom React view over the shared data layer, plus Python media workers.

  • Data: a media schema in the lifeos Postgres (CT 109), workspace-scoped via core.apply_workspace_rls(), in the seeded gus-the-bass workspace. Same pattern as the existing tasks/habits/docs/inbox domains. Clip rows hold metadata + caption + tags + a pgvector embedding + a path to the file. Video bytes never go in the DB.
  • UI: a clips (media) node type in the renderer registry -> a custom React view (gallery, in-browser player, search, filters, collections, set-thumbnail, the fast caption/review mode). clips.justinsforge.com is a door/alias into this module, not a separate program.
  • Python workers (kept, per the stack decision): transcode/proxy + thumbnails (FFmpeg), CLIP/Whisper tagging (ML sidecar), icloudpd ingest. This module is the strongest proof of the "keep Python for heavy data/ML jobs" decision.
  • Build order: build the suite base first (Phases 0-3: repo + data model + JSON API + shell/registry + one simple proof module such as finances or habits). Clips is a later module (Phase 4-ish), NOT the first one. Reserve the media pipeline in the design now so it's planned, not bolted on.

6. Already validated

  • Premiere handoff = FCP7 .xml (xmeml), NOT the Sol Premiere MCP. The MCP is Sol/Claude-Desktop-only and GUI-bound (unreachable headlessly). We generated an .xml with Windows W:\ paths and Justin successfully imported it into Premiere on Windows (two sequences + 10 clips). The handoff is a small generator (forge_premiere_xml_builder); "Send to Premiere" emits a bin .xml into a project's Project/ folder for File > Import. Test artifacts: /mnt/workspace/JustinWieb-VR/2026/2026-06-15_CES-Looking-Back/Project/.
  • Spine readiness verified 2026-06-16: Postgres reachable from Console, pgvector 0.8.0, gus-the-bass workspace seeded, RLS helpers present.

7. Fleet-fact corrections (another bot got these wrong)

  • Premiere handoff: use the validated FCP7 .xml, NOT the Sol MCP.
  • GPU transcode is CT 101 (Plex), not CT 102. There is a single Intel iGPU, already dedicated to Plex; a transcode worker contends with it. Proxy generation should run on CPU off-hours or on Vector's GPU, not bolted onto the Plex iGPU.
  • The ML sidecar does not run on CT 109. CT 109 is a 4-core / 4GB / no-GPU box; it STORES embeddings in pgvector but cannot RUN CLIP/Whisper. Inference runs on Vector's RTX 5070 (via /sshrequest). torch/open_clip not installed yet.
  • MinIO / Redis job queue = over-engineering for now; plain files on NVMe served by nginx + a simple worker loop is enough until there's real pain.
  • CT 109 sizing caveat: fine at Gus's ~10k clips; at the full generalist 100k+ scale the pgvector index wants more RAM than a 4GB box shares comfortably. Bump the LXC or give media its own home when crossing that line.

8. Useful idea worth keeping

Richer auto-captions via a vision-language model (even Claude) over a few extracted keyframes -> human-readable descriptions ("Michael nets a flathead off the boat"), making type-to-search much better. Run on favorites / on demand, not all 10k (cost).

9. Open decisions

  • Suite base: scaffold Phase 0 now vs write the Phase 0-3 base build-plan first (Justin leaning: plan first).
  • Whether to run the icloudpd pull in parallel with the base build (pure data, no UI, gets footage safe and cataloged-ready). Needs Michael's Apple ID app-specific password in ~/.forge-secrets/icloud-michael.env.

[Claude Code]