Skip to content

Clips and Frame.io Clone , Media Apps for CreatorTrack

URL: https://mkdocs.justinsforge.com/memory/plans/clips-frameio-media-app-2026-06-18/

Status: VISION + ARCHITECTURE (pre-implementation). 2026-06-18. Roadmap: CreatorTrack step 4 (apps), after Tier 2 + auth + the files/streaming backend. Builds on prior thinking: [[clip-library-findings-2026-06-17]], [[gus-clip-library-mam-2026-06-15]].

What this is

Two sibling apps inside CreatorTrack, both self-hosted and fully owned (no SaaS, no Immich dependency , owning it is the whole point):

  1. Clips , an Immich-style AI footage library. Search your raw footage in natural language ("baby gus eating minnows"), with AI auto-tagging/captioning + manual tags, then act on the results: send to Premiere, copy to a project folder, or spin up a new project.
  2. Review (Frame.io clone) , export a cut, scrub it frame-accurately, leave timestamped comments, and draw/circle on frames. Likely a separate surface sharing the player + storage.

DECISIONS LOCKED (2026-06-18)

  1. AI vision = hybrid (local model for cheap bulk embeddings + Claude vision for rich captions, selectively). Exact models/where-it-runs decided later; pipeline is built model-agnostic.
  2. Build everything locally, own it. No Immich, no third-party search. Our collections + pgvector + our pipeline. This is the point of the product.
  3. Index in place. Footage stays on the Finn NVMe; we never move/copy originals. We store an index row + thumbnails + embeddings only.
  4. Whole-video search now; moment/segment search later (roadmap). See the get-go decision below so we never have to re-ingest the library to add it.

Architecture

The footage never enters our DB; our core Postgres (CT 109, pgvector already live) stores the INDEX. A clip = a row in a CreatorTrack collection, so the whole library rides the Tier 2 collections/properties/views you're finishing.

Footage offloaded to Finn NVMe (8TB active-video tier)
   │  ingest worker (Python on Finn, idempotent, watches the tree)
   ├─ ffmpeg → poster thumbnail + N keyframes (every ~few sec / on scene change)
   ├─ local vision model → per-keyframe embedding  ──► pgvector (timestamped)
   ├─ Claude vision (selective) → rich caption + suggested tags + suggested rename
   ├─ ffprobe → duration, codec, resolution, fps, capture date, camera
   └─ upsert the clip row (path is the idempotency key)
        │
   Clips app (a collection of clips, gallery view)
        │  query: "baby gus eating minnows"
        ├─ embed query → vector similarity over keyframe vectors (max/aggregate per clip)
        ├─ + structured filters (tags, person, date, camera) via the property system
        └─ ranked clips → gallery grid + hover-scrub preview
        │
   Actions on a selection:
        ├─ "Send to Premiere"  → Premiere MCP on Sol imports the files into the open project
        ├─ "Copy to project"   → copies originals into a chosen/new project folder on NVMe
        └─ "New project"       → name + company → scaffold folders + macros (existing skill)

THE GET-GO DECISION for future segment search (decision 4)

Store per-keyframe, timestamped embeddings from day one (we generate keyframes for thumbnails anyway), not a single whole-video vector. Then: - Now: whole-video search = aggregate (max) the keyframe matches up to the clip. - Later: moment/segment search = cluster the matching keyframes into time ranges , a query + UI change, no re-ingest of the library. Also reserve the schema: a clips row is the parent; a future clip_segments table references clip_id + start/end. Don't build segments now, but the model and the embeddings are ready for it.

What's reused vs genuinely new

Piece Status
Clip index = collection w/ media properties Reuse , Tier 2 collections/properties/views
Semantic search engine Reuse , pgvector live on core Postgres (CT 109)
Tags / people / dates / ratings on clips Reuse , the property registry (multi-select/person/date)
"Send to Premiere" Reuse , Premiere MCP on Sol (97 tools, programmatic import)
"New project" button Reuse , scaffold-project skill (folders + macros) behind a UI button
Comments (for Review) Reuse/extend , Tier 2 inline-comment concept, re-anchored to timestamps
AI ingest pipeline (thumbnail + keyframe embed + caption + rename) NEW , Python worker on Finn
Video streaming endpoint (range requests NVMe → browser) NEW , the "files/streaming backend"
Media components: player + frame-scrubber, clip grid, import buttons, upload mgr NEW , a media Tier 1.5 set
Drawing/annotation canvas (Review) NEW , SVG/canvas overlay on the paused frame

The two apps

  • Data model: clips collection (path, thumbnail, duration, codec, res, fps, capture date, camera, tags[], person[], rating, ai_caption, status) + timestamped keyframe embeddings.
  • Ingest: idempotent Finn worker; manual tagging refines AI output; AI proposes renames you approve (or auto for trusted patterns).
  • Search: natural-language (vector) fused with structured property filters; gallery grid with hover-scrub.
  • Actions: Send-to-Premiere (Sol MCP), Copy-to-project, New-project (scaffold skill). The New-project button is independently useful and could ship first (it's just UI over an existing skill).

App 2 , Review (Frame.io clone)

  • Flow: pick/export a cut → hosted/streamed → reviewer scrubs frame-accurately → timestamped comments → draw/circle/arrow on the frame.
  • Data model: a review (video ref + version) + comments[] each {timestamp, body, author, drawing?}; drawing = vector shapes on a normalized frame canvas.
  • Reuses: the player + scrubber + streaming backend; comments concept. New: the annotation canvas + versioned review sharing.
  • Separate surface, same component + storage foundation.

Dependencies / prerequisites (in order)

  1. Tier 2 (collections/views/blocks) , the substrate. IN PROGRESS.
  2. Auth/login , so the library is real per-user.
  3. Files/streaming backend , the gate: range-request streaming of originals + thumbnail serving from NVMe. (Upload matters less here since footage is offloaded, not uploaded; but thumbnails + cover uploads ride the same backend.)
  4. Media components (player/scrubber, clip grid, import buttons).
  5. Clips app (compose on the framework + the ingest pipeline + search service).
  6. Review app (player + annotation canvas + versioned sharing).

Roadmap / phasing

  • Phase A , New-project button (cheap, standalone): UI over scaffold-project. Ship anytime.
  • Phase B , Clips MVP: ingest pipeline (thumbnail + keyframe embeddings + caption) → clips collection → NL search → gallery → Send-to-Premiere / Copy-to-project.
  • Phase C , Review app: player + frame scrub + timestamped comments + annotation canvas.
  • Phase D (future) , moment/segment search: cluster the already-stored keyframe embeddings into ranges; add clip_segments. No re-ingest (per the get-go decision).

Open specifics (decide before build)

  • Exact local vision/embedding model + host (Finn has no strong GPU; Vector has the RTX 5070); Claude-vision caption budget/cadence.
  • Streaming approach (direct range serving vs an on-the-fly transcode/proxy for heavy codecs).

  • Premiere handoff cross-machine path (app on Console/Finn → Premiere MCP on Sol): trigger route.

  • STATUS 2026-06-18 (files backend, step 3 DONE): migration 0035 app.files (RLS) applied; lib/storage.ts (NVMe hash-subdir, traversal guard, ranged reads); app/api/files/upload (multipart->NVMe->row->{id,url}) + app/api/files/[id] streaming (HTTP Range 206/416, currentUser+RLS gated); FileUpload real XHR uploads w/ progress when workspaceId set; prop-types files normalizes refs->stream URLs (Cell stays string[]). tsc --noEmit clean; committed on ct-files (+0035 on forge spine branch).

  • STATUS 2026-06-18 (media components, step 4 DONE): components/media/* standalone, props-driven (src/thumbnail/previewSrc/clips), backend-agnostic. VideoPlayer (HTML5

STATUS 2026-06-18 , Clips MVP (Phase B foundation) DONE on branch ct-clips

Built the real foundation per this plan, AI deliberately skipped (no embeddings/captions; the vision model is still a deferred decision). Branch ct-clips in worktree forge-suite-wt/ct-clips; 4 commits (+ the migration on the forge spine). npx tsc --noEmit clean for all new code (the only 3 errors are pre-existing creatortrack-logo-white.png module-type errors in marketing/auth files — next-env.d.ts isn't generated because we must NOT run next dev/Turbopack in a worktree; they vanish on a real build).

What shipped 1. Migration 0036_clips.sql (applied via forge_workspace_migrate.py): app.clips (path TEXT UNIQUE = idempotency key, workspace_id, created_by, filename, duration_s, codec, width, height, fps, captured_at, camera, size_bytes, poster_path, tags text[] default '{}', status, created_at) + app.clip_keyframes (workspace_id, clip_id FK ON DELETE CASCADE, t_seconds, image_path, UNIQUE(clip_id,t_seconds)). RLS via core.apply_workspace_rls on both. NO pgvector column , per decision 1+4 (dimension depends on the deferred model); keyframe IMAGES are stored so a future vector(N) migration backfills with no ffmpeg re-pass. 2. Ingest worker scripts/ingest_clips.py (lifeos_admin / trusted ETL): walks the footage tree, ffprobe metadata + ffmpeg poster (640w) + keyframes every ~5s (≤16, 480w) to /mnt/workspace/creatortrack-derivatives/clips/<sha256(abspath)>/, upserts clip + keyframe rows. Idempotent on path; re-runs skip already-ingested clips (poster on disk) unless --force; preserves manual tags on re-ingest. Indexes IN PLACE , never moves/copies originals. No AI. 3. lib/clips.ts (RLS data access + dual on-disk path resolution: originals at root-guarded absolute paths, derivatives traversal-guarded under the deriv root) + routes: GET /api/clips (filename/tag/date filter + tag vocab), PATCH /api/clips/[id] (manual tags), GET|HEAD /api/footage/[id] (Range-streamed ORIGINAL, read-only, auth+RLS-gated, mirrors files/[id]), GET /api/clips/[id]/poster + GET /api/clips/keyframe/[id] (JPEGs). 4. app/(suite)/clips/page.tsx (+ ClipsApp.tsx): composes the existing components/media (ClipGrid posters → open → VideoPlayer streaming via the footage route → Scrubber) + manual tag editor + filename/date/tag filters. Token-only, B&W.

Ingest result (real footage): /mnt/.../2026-06-01_Michael-Moving-Into-Austin/Footage , 65 videos. 60 ingested (status=ready), 396 keyframes, 60 posters (16 MB derivatives), all 60 with codec/res/fps/captured_at/camera. 5 failed , source files A999_* are owned by root mode 0600 (unreadable by our user) → rows written with status='error'. Target workspace_id=12 (Justin's Workspace), created_by=1.

INTEGRATOR / follow-up steps (NOT done here)

  • Sidebar nav link → /clips: add to components/workspace/* (the shell, owned by another worker) , intentionally untouched.
  • 5 permission-blocked originals: chmod/chown the root:justinwieb 0600 A999_* files to readable, then re-run ingest (idempotent) to flip them error→ready. Until then they neither index nor stream.
  • Deferred embedding migration (Phase D groundwork): once a vision model is chosen, add embedding vector(N) to app.clip_keyframes + a backfill job reading image_path off disk , no ffmpeg re-pass (the get-go decision). Then add NL/vector search to lib/clips/GET /api/clips and app/(suite)/clips.
  • Deferred AI ingest: wire local-embedding + selective Claude-vision caption/tag/rename into ingest_clips.py (hooks are isolated; metadata + frames already extracted).
  • Actions (Send-to-Premiere / Copy-to-project / New-project): not built; ClipGrid already supports multi-select for a future action bar.
  • Ops: re-run ingest on a cron/watch for new footage; env knobs CREATORTRACK_DERIVATIVES_DIR, CREATORTRACK_FOOTAGE_ROOTS (default /mnt/workspace).
  • next-env.d.ts: generated on first real next build (don't next dev in the worktree , Turbopack panics); clears the 3 pre-existing PNG tsc errors.