Skip to content

Handoff: CreatorTrack News hub (Google-News-style briefing + archive.ph) (2026-07-05)

Branch: feat/news-hub (worktree ~/forge-suite-wt/feat/news-hub, dev slot :3066, DB ct_feat_news_hub). NOT merged, NOT deployed. 8 commits on top of main (41ca4f2..ebb04f3).

Round 5 (same day): every paywall-toolkit row now offers two actions (the user's ask): Extract (pull it through that one method on our server, render inline) and Open (new tab). Extractable methods = Archive.today, Wayback, Jina, Googlebot (services that block our datacenter IP stay Open-only). New GET /api/apps/rss/article?method=<m> single-strategy path (getReadableArticleVia in news-article.ts); load() in ArticleReader threads a method. Jina rewritten to default markdown mode (parses Title:/Markdown Content:, markdown images as inline blocks, nav/boilerplate + nav-list heuristic filters) so title + body are clean where raw-text mode leaked site chrome; extractReadable rejects nav-junk titles so the feed's own title wins.

Round 4 (same day): ad/sponsored feed-item filter (lib/news-adfilter.ts: host denylist + lead-gen title regex; CNN injects lendingtree.com lead-gen forms as feed items) applied at ingest + prunes stored rows on refresh; paywall toolkit (lib/news-bypass.ts + BypassToolkit in ArticleReader) = "Try another way to read it" expander with 8 browser-side bypass services (archive.today submit/view, Wayback + Save Page Now, Jina, 12ft, RemovePaywall, Textise, Freedium/Medium), each opens in the user's browser where their IP/session works; Jina Reader added as server-side step (renders JS) between Wayback and archive.today; boilerplate strip (cookie/TOS/subscribe/video-playlist scaffolding) in the extractor. Bloomberg remains hard (many items are VIDEO pages with no article body; toolkit is the answer there).

Round 3 (same day): shared ArticleReader component gives the Feeds tab's ReadingPane the same full-text extraction as the briefing overlay; sidebar filter-sources box; resizable list/pane split (shared ResizeHandle, persisted); extraction quality pass (best-<article>-by-text scoping kills related-story title leakage, link-density junk filter, inline images as ordered blocks, og-image dedupe); Google News shim links (news.google.com/rss/articles/<id>) decoded via the DotsSplashUi batchexecute RPC (same flow as googlenewsdecoder; sg/ts from the article page's data attributes) so Reuters/AP/Athletic extract + archive against the real URL and read-later saves store it. Note: server-side extraction never executes JS, so it inherently does the "toggle JS off" paywall trick; thin-content detection catches JS-rendered articles and escalates to Googlebot/Wayback/archive.

Round 2 (same day): catalog grown to 139 outlets with new Opinion (12) and Reads/longform (14) categories, +9 tech, Reuters/AP/The Athletic via Google News site-search RSS proxy (news.google.com/rss/search?q=site:reuters.com...), Defector/Semafor/Quartz/IGN/Polygon etc; all 56 additions live-validated, 0 sync errors. Paywall chain upgraded: original -> Googlebot UA -> Wayback Machine -> archive.today mirrors with per-host 10-min 429 cooldown (verified: FT article resolved via the archive path after archive.ph 429'd; brand-new hard-paywalled NYT pieces with no snapshot remain link-out-only). Briefing gained a search-results grid (header search box now works on both surfaces); reader aggregate views use source-diversified ordering so CNN's posting frequency can't wallpaper the top (single-feed + search views stay chronological).

What was built

News app (/rss) grew a second rail surface modeled on Google News, plus archive.ph article access:

  • Curated catalog lib/news-catalog.ts: 83 outlets across top/world/us/business/technology/entertainment/sports/science/health + local-austin (7), local-texas (5), local-omaha (4). Every feed URL validated live through the app parser on 2026-07-05 (syncCatalog returned 0 errors). One-click sync op, idempotent on (workspace, url); removed feeds stay removed until manual re-sync.
  • Briefing surface (NewsBriefing.tsx, default view): category chips, Your briefing masthead, Top stories hero cluster, per-category card rows (source-diversified: one item per feed first), Local news box with Austin/Texas/Omaha chips (own API slice so quiet local feeds never fall out of the latest-500 window). Classic three-pane reader remains as the second rail surface.
  • archive.ph integration (lib/news-article.ts + /api/apps/rss/article): reader mode extracts full text server-side, original URL first, archive snapshot fallback when paywalled/thin (<3 paragraphs). Mirror rotation archive.ph -> archive.today -> archive.md with browser UA; 404 anywhere = "no snapshot yet" (authoritative, offers submit link), 429/5xx = next mirror. In-process cache 1h/300 entries. Full mode = https://archive.ph/newest/<url> link-out buttons in overlay + classic reading pane.
  • Read Later: unchanged single source of truth (saveArticle -> knowledge readlater node). New: archive-snapshot fallback via saveArticle(..., { prefetched }) when the original won't extract; note always stores the real article URL.
  • Schema 20260705T201500_news_hub_catalog.sql: rss_feeds + category, catalog_id, homepage, last_fetched_at (+ index). Applied to dev clone only; prod needs forge_workspace_migrate.py at deploy time.
  • Refresh: staleness gate (15 min) + concurrency pool (8) in refreshAllFeeds; manual Refresh forces.

Verified (live, dev slot user 79)

  • syncCatalog: 83/83 outlets, 500 items cached, all 12 categories populated.
  • Reader overlay: BBC + Verge full extraction (16 paras, hero image). Playwright screenshots of briefing home, overlay, Technology category all good.
  • saveToReadLater: created knowledge node 5736 in readlater section.
  • Archive fallback: mirror walk works; archive.today answered 404 (=no snapshot) where archive.ph 429'd. Caveat: all mirrors were 429-throttling Console's IP by end of session, so server-side archive extraction is intermittent; UI degrades to error + retry + direct links (which work in the user's browser).

Known rough edges / next

  • archive.today throttling: consider a per-host cooldown + honoring first-hit only, or routing through the browser-stream box if it becomes chronic.
  • Feeds with dead image URLs hide thumbnails (onError) rather than proxying images.
  • folder on catalog feeds is set to the category label so the classic FeedSidebar groups sensibly; renaming a category in the catalog re-stamps on next sync.
  • Teardown when done reviewing: forge_creatortrack_agent_teardown.sh feat/news-hub (or chat.justin Dev slots button).

URL: https://mkdocs.justinsforge.com/memory/handoffs/creatortrack-news-hub-briefing-2026-07-05/