Skip to content

Reference mkdocs build pipeline

URL: https://mkdocs.justinsforge.com/memory/general/reference_mkdocs_build_pipeline/

Pipeline

  • Source config: infra/mkdocs/mkdocs.yml (docs_dir: ../../, i.e. the whole forge repo minus exclude_docs; site_dir: /home/justinwieb/forge-mkdocs-site).
  • forge-mkdocs-build.timer fires forge-mkdocs-build.service every 2 minutes (OnUnitActiveSec=2min). ExecStartPre runs scripts/forge_mkdocs_preflight.py (auto-repairs bad UTF-8, always exits 0); ExecStart runs scripts/forge_mkdocs_build_atomic.sh.
  • forge-mkdocs.service is a bare python3 -m http.server 8000 --directory /home/justinwieb/forge-mkdocs-site --bind 0.0.0.0, tunneled to https://mkdocs.justinsforge.com behind Cloudflare Access.
  • Unit files live at infra/systemd/forge-mkdocs*.{service,timer}; kept synced to /etc/systemd/system/ by hand (no auto-deploy step yet, sudo systemctl daemon-reload after edits).

2026-07-09: systemic 404 bug, root cause + fix

mkdocs build defaults to --clean, which deletes site_dir before regenerating it. The build previously ran that command directly against the live forge-mkdocs-site dir that forge-mkdocs.service served from, so every ~45s build opened a window (roughly a third of all wall-clock time, since the timer fires every 2 min) where the docroot was empty or partial. Any request landing in that window 404'd, for old published pages just as much as freshly committed ones. This is what "mkdocs always lags or 404s" was.

Fix (scripts/forge_mkdocs_build_atomic.sh): build into an offline, double-buffered scratch dir (forge-mkdocs-site-a / -b, alternating), then atomically repoint the forge-mkdocs-site symlink at the freshly built one via mv -T (single rename(2), no partial-state window). forge-mkdocs.service must use --directory <symlink> rather than WorkingDirectory/chdir, because chdir() resolves a symlink once at process start and binds to that physical inode forever; --directory makes http.server store the literal symlink path and re-resolve it fresh on every request, so it picks up each swap with zero restart and zero downtime. Verified via continuous polling through a full build cycle: 0 non-200 responses.

Publish latency (current, expected going forward)

A commit is live within one timer period: up to ~2 minutes, typically less. This did not change with the fix, only the flakiness did, previously-published pages no longer 404 during someone else's rebuild. If you need a doc visible immediately (no wait), use SendUserFile (render) instead of a bare mkdocs link, see remote-control-doc-delivery.

Verifying externally

https://mkdocs.justinsforge.com/... is Cloudflare Access-gated; an anonymous curl gets a 302 to the CF Access login, which is the expected healthy response (proves tunnel+origin alive, matches the ACCEPTED_STATUS convention in scripts/forge_uptime_kuma_provision.py). To check real content, curl the origin directly: curl http://127.0.0.1:8000/<path>/ on Console.