Skip to content

Reference creatortrack scripts app

URL: https://mkdocs.justinsforge.com/memory/general/reference_creatortrack_scripts_app/

Built on feat/social-data-layer-and-scripts (plan memory/plans/creatortrack-social-data-layer-and-scripts-app-2026-07-14.md, Tasks 12-18). Not yet merged to main or deployed to prod as of this writing; ships alongside Social, see reference_creatortrack_social_app.md's Ship checklist.

What it is

Scripts is a route app (no node face) that is the consumer in the Social/Scripts split: it reads app.social_videos (transcripts, structure, embeddings) but never touches a platform API and never generates video content itself, only text scripts. It owns app.voice_profiles and app.generated_scripts.

Schema

app.voice_profiles: id, workspace_id, name (conflict key (workspace_id, name), enforced in application code only, no DB unique constraint), profile jsonb ({stats, card}), source_video_count, generated_at, model.

app.generated_scripts: id, workspace_id, voice_profile_id (FK ON DELETE RESTRICT, deliberately not CASCADE: deleting a voice profile must never silently wipe its generated scripts and their source_video_ids audit trail), topic, platform_target (CHECK youtube/instagram/tiktok/facebook, duplicated verbatim from social_videos.platform's CHECK list, a known future single-source-of-truth cleanup candidate), body jsonb ({hooks[], beats[], cta, shot_list[]}), source_video_ids bigint[] (the actual retrieval pool used for that generation, no FK, intentional audit trail), task_node_id/page_node_id (both real FKs to app.nodes(id) ON DELETE SET NULL, matching this repo's established task/page-reference convention).

app.social_videos.embedding vector(384): added here (not Social's own scope) since it exists purely to serve Scripts' retrieval. bge-small-en-v1.5 via @huggingface/transformers (local ONNX, no network calls at inference, same model forge's own /recall semantic search uses via fastembed's Python wrapper of the identical weights). No ANN index (IVFFlat/HNSW) at today's ~515-row-per-workspace scale; exact <=> scan is sub-millisecond.

The "voice profile, not a fine-tune" decision

Deliberate: the corpus (497 verified transcripts after quarantining 18 junk rows, ~140k tokens) is both too small and too narrow (one creator, one channel) to fine-tune a stable, generalizable model on, and a fine-tune would be opaque and unfixable by the creator. Instead: retrieval-grounded few-shot generation, prompt-injecting a hand-editable two-layer voice card (profile.stats deterministic aggregation, profile.card LLM-synthesized markdown prose a creator can hand-edit in a future Settings UI). Hand-editability is the explicit product advantage over a fine-tune; lib/scripts/regenerate.ts splits "recompute stats only" from "resynthesize the card" (destructive, needs an explicit in-app confirm modal, never window.confirm) specifically to protect hand edits from being silently overwritten.

Generation flow

generateScript(id, workspaceId, topic, {platformTarget?, voiceProfileId?}) in lib/scripts/generate.ts is the single orchestration entry point; both the API route (POST /api/apps/scripts/generate) and the chat tool call it directly, never re-implement retrieval or the LLM call. Two-stage retrieval: cosine-rank the whole workspace corpus by embedding <=> query, take top 15, then re-rank that pool by real retention (average_view_percentage DESC NULLS LAST, never coalesced to 0), take top 5. Prompt = profile.card (verbatim) + a small stats extract (not the full ~4-6KB profile.stats blob) + the 5 retrieved exemplars + topic/platform. Strict JSON output, one retry on malformed JSON, em/en-dash sanitized. ~34s p50 latency (all Sonnet generation time, no streaming this round). Deterministic given the same topic string (same embedding in, same ranked pool out).

Chat tool

generate_script in lib/chat.ts's TOOL_DEFS, calls generateScript directly (no HTTP hop). Ungated: no tool in this chat registry is gated on installed apps today (a repo-wide fact, not a Scripts-specific gap); adding real per-app gating would be new cross-cutting work. listVoiceProfiles (new) resolves an optional voice_profile name argument. Zero-voice-profile workspaces short-circuit before any paid LLM call. Low-similarity caution threshold 0.7 (below this, tells the user the retrieved examples were a weak topical match) is currently duplicated as a literal in 3 places (UI, chat tool, doc comment), no shared constant yet.

EVAL VERDICT (verbatim honest, fair-baseline run)

PASS. Judge win rate 100% (17/17 scored). Mean embedding similarity delta +4.9 percentage points. Positive-delta videos 12/17 (70.6%). This is the corrected, fair-baseline number: an earlier run's baseline arm was missing the same JSON field-semantic prompt guidance the profiled arm had, which inflated the original (superseded) delta to +5.2pp and the positive-delta rate to 87.5%. The qualitative signal (judge win rate) held at 100% through every configuration tested, including before and after the fair-baseline fix; only the noisier embedding-similarity metric moved.

Franchise-template caveat, record this alongside the verdict: Gus The Bass's back catalog is heavily built on recurring numbered franchise formats ("will my pet bass eat X" up to part 14+, annual weigh-ins, "crawfish squid games"). When a holdout topic matches an existing franchise, retrieval surfaces a near-literal structural template to imitate, which is a legitimate, intended product win (the whole point of retrieval-grounded generation) but means part of the measured lift reflects template transfer, not a pure voice signal that generalizes equally to a genuinely novel topic with no retrievable precedent. Expect the strongest lift on franchise-matching topics, a smaller (though still positive, per this run) lift on novel ones.

Production-path reliability, favorable finding: the fair-baseline re-run measured 0/20 generation-arm failures (both profiled and baseline succeeded on every holdout video); all 3 residual failures (15%) were isolated to the eval's own judge-step JSON parsing (lib/scripts/eval.ts, not shared with production generate.ts). This is one run's worth of evidence, not a guarantee, but it materially tempers an earlier "~25% residual production risk" framing.

Gotchas

  • profile.card is a plain markdown string, ready for direct prompt injection; profile.stats is structured JSON meant for the profile-synthesis pass itself, not for dumping whole into a generation prompt (too large, ~4-6KB).
  • upsertVoiceProfile overwrites profile.card unconditionally on a full regenerate. The stats-only vs. destructive-resynth split (see above) exists specifically to prevent silently discarding a creator's hand edits.
  • extractJsonObject is now 7 copies repo-wide, resolveWorkspaceId is 5 copies across /api/apps/* routes; both are pre-existing repo conventions (small duplication over cross-feature coupling), flagged as future single-source-of-truth candidates, not fixed in this plan.
  • platform_target/platform CHECK constraint is duplicated verbatim between app.social_videos and app.generated_scripts; a fifth platform would need updating in two places.
  • NoExemplarsError-style typed error for the chat tool's read-path profile-name ambiguity check was carried as a nit, not built; worth doing alongside any future touch of that surface.
  • Whisper transcription is not wired (no local whisper binary/GPU on Console); relevant to Scripts because a bigger transcript corpus (whisper-fallback videos) would directly improve profile/generation quality.