Reference asset grabber

Full playbook: /home/justinwieb/forge/docs/asset-grabber-playbook.md.

Stack

• Python venv: ~/.forge-venvs/assets/ (outside git repo). Rebuilt fresh 2026-06-10 after corruption (see playwright-fix-2026-06-10 handoff); package set: playwright==1.58.0 pillow gallery-dl requests + playwright install chromium.
• Playwright, launched with channel="chromium" (full Chromium, NOT the headless shell; the shell trips engine bot detection).
• cwebp, avifenc, yt-dlp, gallery-dl, Pillow, ffmpeg, imagemagick, exiftool

Scripts (flat at scripts/, wrapper forge_assets_run.sh)

Script	Purpose
forge_assets_grab.py <url>	Scrape a page for images/videos. Handles lazy-loads, downloads through browser context (cookies + referer).
forge_assets_search.py <query> --engine {google,bing,ddg,unsplash,pexels}	Image search. Default engine bing (2026-06-10: google/ddg serve anti-bot walls headless). API for Unsplash/Pexels (optional UNSPLASH_ACCESS_KEY/PEXELS_API_KEY env).
forge_assets_optimize.py <path>	WebP + AVIF at responsive widths (default 320/640/1280/1920 + source width). Strips EXIF by default.
forge_assets_catalog.py	Scans /mnt/workspace/Assets + forge/assets → forge/data/assets-catalog.json. Includes sha256, dims, tags, and any source_url pulled from sibling provenance.json.
forge_assets_run.sh {grab\|search\|optimize\|catalog}	Bash wrapper that invokes the venv python.
assets/lib/provenance.py	Writes/merges provenance.json. Pure trace, NOT a license gate. Scripts add scripts/assets/ to sys.path for it.

Default output locations

• Grabs: /mnt/workspace/Assets/Web-Grabs/<date>_<host-or-query>/
• Optimized for a specific site: forge/sites/<site>/<page>/assets/
• Catalog: forge/data/assets-catalog.json

Licensing stance

None. Any image/video/audio on the web can be pulled. provenance.json exists only so we can trace which page something came from, not as a permission check.

Auth'd sites

Use a persistent Chromium profile at ~/.forge-venvs/assets/browser-profile/ (outside repo). First login headful, subsequent headless. Never commit the profile.

Rate limit

Default 250ms between downloads inside a single run. Bump time.sleep(0.25) in the script if a specific site needs slower.

Known behavior

• 2026-06-10: google + ddg scrape engines blocked by engine-side anti-bot (DDG escalates to an anomaly/CAPTCHA page on repeated hits). Use bing or the API engines; not pursuing evasion.
• Google Images (when not walled): script clicks thumbnails to upgrade to full-res. Slower but produces real assets.
• Wikipedia pages: script filters out /wiki/File:* HTML pages, keeps actual upload.wikimedia.org direct image URLs.
• optimize.py emits source-width variant plus any targets below source width. Won't upscale.