Skip to content

AI Video Editor Pipeline, Vision & Architecture

URL: https://mkdocs.justinsforge.com/memory/plans/ai-video-editor-pipeline-vision-2026-06-20/

Date: 2026-06-20. Author: [Claude Code], with Justin. Status: vision + architecture, pre-build. Build order and MVP defined; Phase 1 not yet planned in detail (next step: /feature-plan). Read alongside: - Clip library findings: clip-library-findings-2026-06-17 - Suite architecture: workspace-suite-architecture-and-rebuild-plan-2026-06-17

1. The north-star goal (Justin's words)

Claude has access to all of our videos, knows what each one is and how we edit, can go through and take voiceover, cut it down, get clips, edit them, then render that or lay it out in an Adobe project so we can start programming it.

2. The key reframe (what makes this buildable)

This is NOT "build AI video editing" as one magic capability. It decomposes into a structured-data pipeline where Claude's only job is to emit an edit decision list (EDL): which clips, in what order, cut at which timecodes, with which audio/VO. Everything downstream of that decision is deterministic plumbing (FFmpeg, project-file generators). The "Adobe project" is just the final serialization format the EDL gets written into.

This matches the Forge philosophy: intuition becomes machine execution; capture is fast, processing is deterministic, observability is mandatory.

3. The capability stack

Layer What it does Buildable today? Runs on
1. Index & understand Every clip transcribed (Whisper word-level), scene/shot cuts detected (PySceneDetect), silence detected, CLIP embeddings, VLM captions. "Claude knows what each video is." Yes, this IS the clip library already in progress Finn + spare-PC GPU
2. Editorial brain Given a goal ("60s short of the flathead catch"), Claude reads transcripts + scene data + style notes and emits a structured cut list: clip IDs, in/out timecodes, order, which audio/VO. Yes, structured JSON output is Claude's strength Claude API
3. Voiceover Use existing VO (from transcript) OR generate new VO via TTS (HyperFrames TTS skills already available). Yes Finn / API
4. Assemble & output Cut list to FFmpeg render (simple) OR native .prproj (scripted Premiere) OR the FCP7 .xml already generated. Yes; .xml done, .prproj via spare PC Finn (FFmpeg) / spare PC (Premiere)
5. Learn how we edit Feed Claude past finished edits (source + final cut) so it learns pacing, what gets kept, hook structure. Few-shot + a living style doc. Partially, the hard iterative part Claude API

4. The honest constraint: Layer 5 is the hard one

Claude can nail a rough cut quickly (remove dead air, pick the moments with the key words/action, order them, drop in VO, hand over a ~70%-done timeline). That alone is a large time-saver and is fully buildable now.

The finesse (J/L cuts, music-driven pacing, color, motion graphics, emotional beats) is where Justin/Michael still finish by hand. So the realistic v1 target = Claude builds a rough cut as a real Premiere project; a human finishes the last ~30%. NOT "render a finished video untouched" on day one. The system climbs toward hands-off as Layer 5 improves.

5. File-format reality (why .xml now, .prproj later)

  • FCP7 .xml (xmeml) is an open, documented interchange format Premiere can IMPORT. Already generated by forge_premiere_xml_builder (pure Python, no Adobe). It is NOT a project file; you still open Premiere and File > Import. Validated working on Windows.
  • .prproj is Adobe's proprietary, undocumented, version-specific project format. It cannot be reliably hand-authored from Python. The only thing that produces a valid .prproj is Premiere Pro itself, scripted via ExtendScript/UXP (create project, import media, build sequences, save). This is the one capability the .xml generator, Finn, and a Linux box cannot provide, and it is required for native double-click projects and a fully hands-off (project to AME render) pipeline.

6. Hardware roles (where each layer runs)

Finn (MS-01, Proxmox)

  • i9-13900H (20 threads), 125GB RAM (~80GB genuinely idle, no ZFS so no ARC win).
  • Single Intel Iris Xe iGPU, already dedicated to Plex (CT 101) via renderD128. Do NOT bolt a transcode/ML worker onto it; it contends with Plex.
  • Can add a low-profile, single-slot, <=75W GPU (Arc A380 / RTX A2000) via its PCIe x16 (x8 electrical) slot. Only worth it for AV1 or local AI; Quick Sync already covers Plex.
  • Role in pipeline: CPU off-hours proxy/render batches, indexing orchestration, the data spine (forge-data CT 109, Postgres + pgvector).

Spare PC (the Adobe appliance) , decision pending

  • AMD Ryzen 5 2600 (6c/12t), GTX 1660 Super (full Turing NVENC: H.264 + HEVC encode/decode, the real encoder, not the crippled GTX 1650 one), 16GB DDR4, 500GB SSD, Asus PRIME B450M-A.
  • Idles ~120W, so ~$12-15/mo if run 24/7 at Austin rates. Wake-on-LAN / on-demand only; do not run as an always-on server.
  • The 1660S is effectively a free RTX A2000 equivalent for this scale, so it removes the GPU-purchase objection and the Finn-VM fragility objection at once.
  • Linux vs Windows decision:
  • As a Linux render/ML node it unlocks nothing new: proxies (Finn CPU off-hours, free), CLIP/Whisper (Vector RTX 5070, stronger), FFmpeg renders (Finn) are already covered. Paying 120W to relocate existing work is a bad trade.
  • As a Windows + Premiere box it unlocks the genuine new capability: scripted native .prproj generation + hands-off AME render (Layer 4's top tier). This is the only justification for the box, and it is real.
  • Recommendation: stand it up only if native .prproj/hands-off render is wanted; run it wake-on-demand. Costs a CC activation slot (Vector + Sol may already be the 2).

Vector (RTX 5070, 9950X, 64GB, Win11)

  • Interactive editing workstation. Hero edits finished here by hand.
  • On-demand CLIP/Whisper inference via /sshrequest (per clip-library findings; CT 109 only STORES embeddings, cannot run ML).

Claude API

  • Layers 2 and 5 (editorial brain + style learning). Cost-aware: VLM captions and editorial passes run on demand / on favorites, not blindly across all 5-10k clips.

7. Encode / decode / transcode primer (for reference)

  • Encode: compress raw video into a codec (H.264, HEVC/H.265, AV1). Camera/editor export.
  • Decode: uncompress a codec into displayable frames. The playback device does this.
  • Transcode: decode then re-encode into a different codec/resolution/bitrate. The server does this on the fly. "Hardware transcode" runs on Quick Sync/NVENC, not the CPU.
  • Codec vs resolution are separate dials. HEVC to H.264 is a codec change (required for browser playback); 1080p to 720p is an optional size/bandwidth choice. You do NOT have to drop resolution to make HEVC play in a browser.

8. HEVC / web-playback decisions (clips app)

  • Browsers cannot reliably decode HEVC (Safari can; Chrome/Firefox/Edge mostly cannot). iPhone footage is HEVC; iPhone photos are HEIC (same problem).
  • Never serve the original HEVC to a browser. On ingest, generate web derivatives and serve those; the HEVC original stays on the NVMe and is what gets sent to the project.
  • Per clip generate three derivatives:
  • Poster thumbnail (JPEG/WebP) for the gallery grid.
  • H.264 MP4 proxy (faststart, ~720p) for in-browser play + scrub. 720p is a deliberate proxy choice for fast scrub/cheap streaming, NOT an HEVC requirement.
  • Storyboard sprite sheet (JPEG grid every ~5-10s) for hover-scrub filmstrip preview.
  • Do not transcode in the browser. ffmpeg.wasm / WebCodecs are slow, unreliable, and re-do the work every view. Transcode once server-side, cache the proxy, serve forever.
  • Proxy generation for the 5-10k backlog is a one-time batch: feasible on Finn CPU off-hours (one-time job measured in hours/overnight) or fast via the spare-PC NVENC. Benchmark on ~20 real clips first (probe-first time estimates).

9. AI renaming / naming

  • Do NOT rename files on disk (breaks the icloudpd manifest + idempotent re-ingest, and loses original date/filename metadata). Store the AI description as the title/caption column in Postgres; the name shown/searched is a DB field, not the filesystem name.
  • CLIP tags every clip (fast on GPU). Human-readable names ("Michael nets a flathead off the boat") come from a VLM over a few keyframes, run on favorites/on-demand (cost-bounded).
  • For Premiere bin names, generate descriptive names at export time from the DB caption into the .xml/.prproj. Files on disk stay IMG_1234.mov.

10. Build order / MVP

  1. Clip library indexing (Layer 1) , in progress, hard prerequisite. Cannot edit what is not indexed.
  2. One-video proof: Claude reads a single video's transcript + scene cuts, proposes a 60s short as a cut list, emits the existing FCP7 .xml, Justin imports to Premiere and judges it. No new infra required for this step.
  3. Iterate cut quality (Layer 5): build the living style doc; few-shot from past edits.
  4. Automate Layer 4 output: native .prproj + optional auto-render on the spare-PC Adobe appliance (only if the .prproj/hands-off path is chosen).

11. Open product decisions (settle before Phase 1)

  • v1 target: rough-cut-for-human-finish vs fully-finished-untouched render. (Recommend rough-cut first.)
  • Which brand/footage first (Gus The Bass is the clip-library pilot).
  • Voiceover: use existing VO from clips vs generate new VO via TTS.
  • Spare PC: Linux (no new capability) vs Windows+Premiere (native .prproj); only build the Windows appliance if native projects / hands-off render is the goal.

12. Next step

Run this through /feature-plan to pin the open decisions above and write a numbered Phase 1 build plan (exact file paths, verification gates, commit boundaries) to memory/plans/.