project creatortrack social transcript backfill terminal 20260715T111619

As of 2026-07-15, a multi-day rate-limit-constrained backfill was pulling transcripts for all 548 of Justin's YouTube videos into forge/data/creatortrack/social-transcripts-ws85.jsonl, gated by YouTube's anti-scrape throttling (mitigated partway through with an EJS challenge-solver flag on the ingest worker).

Why this is worth remembering: the official Captions API was ruled out (needs an unheld youtube.force-ssl OAuth scope, and ~200 quota units/video would blow the 10k daily quota across 219 remaining videos at the time); VPN/second-IP and cookie-based scraping were both ruled out as not worth the risk/complexity for a one-time job.

How to apply: this full scrape should never need to run again. Once it converges, treat the jsonl export as the source of truth for existing video transcripts; any future need (new dev slot, prod deploy, extraction/profile tasks) should read from that file. Only newly-published videos (a handful per week) need fresh per-video scraping going forward. If a future session considers re-running a full backfill, that's very likely unnecessary, check the export file first.

[auto-memory session 60d132de-4659-485c-8044-427af0ccce7f, confidence 0.65, mode staged]