feedback disposable store expensive data export 20260715T101942

---

name: Export expensive-to-acquire data at creation time, not after description: Lesson from the 2026-07-15 CreatorTrack transcript-loss incident: a clean git tree only certifies code is safe, never data; costly-to-reacquire data written to disposable/dev-scoped storage needs an explicit export step in the same job that produces it. type: feedback

---

Incident: a CreatorTrack dev-slot DB (disposable, prod-clone, excluded from forge-backup) held ~6 hours of rate-limited YouTube transcript scrapes. The reaper's data-loss guard is git status — clean tree means "nothing to lose." The worker had committed all code (as doctrine requires), so the guard passed and a full teardown dropped the DB along with it, destroying the only copy of the transcripts. Only run logs survived; recovery required a full re-scrape.

Why: the assumption baked into the dev-slot design was "disposable DB contents are re-derivable, who cares if they die." That's true for cheap-to-fetch data (e.g. video metadata, minutes to re-pull from an API) but false for expensive-to-fetch data (rate-limited scrapes, hours of compute). Nobody priced that difference when the ingest was written, so the expensive output silently inherited disposable storage. Git cleanliness measures code risk, not data risk — the two got conflated.

How to apply: before landing any ingest/scrape/backfill job that writes into a dev-scoped, disposable, or non-backed-up store (dev-slot DBs, ephemeral worktrees, temp caches), ask whether the source data is cheap or expensive to reacquire. If expensive, add an export step (flat file to durable disk, e.g. forge/data/<project>/...jsonl) that runs at the moment the job converges, not as an afterthought. This generalizes beyond CreatorTrack to any pipeline landing scraped/rate-limited/paid-API data in a dev or disposable data store.

[auto-memory session 60d132de-4659-485c-8044-427af0ccce7f, confidence 0.75, mode staged]