Skip to content

URL: https://mkdocs.justinsforge.com/memory/general/reference_audiobook_download_workflow/

Audiobook + ebook automation workflow

As of 2026-06-07 this is automated, not manual. Three layers on CT 102 (media-server), all in the gluetun netns:

Layer Role (movie-stack analogy) Surface
shelfarr Readarr + Overseerr for books (search → grab → import) shelfarr.justinsforge.com (CF Access gated)
Audiobookshelf Plex for audiobooks (viewer only, auto-scans library) audiobooks.justinsforge.com
calibre-web (CWA) Plex for ebooks + auto-importer + Send-to-Kindle calibre/CWA UI (port 8083)

shelfarr (the automation engine)

  • Image ghcr.io/pedro-revez-silva/shelfarr:latest — actively maintained (do NOT confuse with Readarr, which Servarr archived in 2024).
  • Wired to Prowlarr (indexer hub) + qBittorrent (download client, localhost:8080 via shared netns).
  • Sources: MAM (premium, Prowlarr indexer id 12) plus built-in direct sources — Anna's Archive, Z-Library, LibriVox, Project Gutenberg, Open Library/Hardcover metadata.
  • Mounts: /audiobooks, /ebooks (=/mnt/storage/books), /downloads. Config (sqlite) at /opt/stacks/media/shelfarr/production.sqlite3.
  • Use it instead of browsing MAM by hand: open shelfarr, request a title, it searches sources → grabs via qBit → lands in the right library.

Exposure fix (2026-06-07)

shelfarr was unreachable for a while: gluetun published 5056:80 but shelfarr listened on 3000, so host:5056 hit a dead port. Fix = added PORT=80 to shelfarr's compose env so Puma binds the already-published netns port 80. No gluetun recreate needed (recreate only shelfarr; it rejoins the existing netns). Then /cf-add shelfarr justinsforge.com http://192.168.86.74:5056 + a CF Access app ("Justin (secure)", emails [email protected] + [email protected], 24h). If shelfarr ever 404s at the edge, check it's still binding 80: docker exec gluetun netstat -tlnp | grep :80.

Ebooks (CWA auto-ingest)

  • crocodilestick/calibre-web-automated:latest. Drop any ebook into /mnt/storage/downloads/books (/cwa-book-ingest) → CWA auto-converts + imports into the calibre library at /mnt/storage/books. Processed-book audit dirs under /opt/stacks/media/calibre-web/processed_books/.

Kindle (Send-to-Kindle via CWA)

  • CWA emails books to Justin's Send-to-Kindle address. Archiving is ON, so a send lands in the Amazon cloud library and syncs to every device/app (Kindle, phone Kindle app, iPad, Mac, web) via Whispersync. Send once, read anywhere.
  • SMTP: smtp.gmail.com:587 STARTTLS, from/login [email protected], Gmail app-password stored encrypted in CWA app.db (mail_password_e, Fernet via /config/.key). Backup copy: media-server:/root/.forge-secrets/cwa-mail.env.
  • Amazon approved sender: [email protected]. Kindle address: [email protected] (also iPad/Mac addresses exist; not needed since archiving syncs).
  • To change the password: encrypt with CWA's .key then write mail_password_e (raw plaintext write breaks decryption); restart calibre-web. Gotcha 2026-06-07: a stale 19-char password had silently broken all sends.

Audiobookshelf library folders

ABS scans three roots; shelfarr/manual drops go under the right one:

Folder What
/mnt/storage/audiobooks/Justin Audiobook/ Justin's
/mnt/storage/audiobooks/Kristine Audiobooks/ Kristine's
/mnt/storage/audiobooks/Personal Collection/ shared/niche

Convention: Author - Title/ with mp3 or m4b inside; ABS scrapes Audible/Google Books at scan time.

Gotchas

  • If shelfarr/qBit/Prowlarr unreachable: gluetun likely recreated → network_mode: container:gluetun siblings stranded. Fix: docker compose -p media-server up -d on CT 102 (note the -p media-server project-name quirk).
  • Permissions: files dropped into a library must be readable by the container UID (PUID=0 here).

How a request flows (UX)

Request a book → shelfarr searches all indexers (MAM via Prowlarr + Anna's/Z-Lib/LibriVox/Gutenberg) → a result must be selected → a queued Download is created → DownloadJob dispatches to qBit → import. auto_select_enabled defaults to false, so requests sit at status "searching" until you open the request and pick a result in the UI. Set auto_select_enabled=true (settings table; min_seeders + confidence_threshold 90 guard it) for hands-off Radarr-style auto-grab. Job classes: request_queue_job, search_job, download_job (perform(download_id)), download_monitor_job, post_processing_job.

Troubleshooting (fixed 2026-06-07)

  • Symptom: requests search fine but never download; logs spam [Qbittorrent] Failed to fetch diagnostics: ActiveRecord::Encryption::Errors::Decryption. Cause: the qBit client password was stored as plaintext (Success402!) from an older shelfarr build; the current build expects it AR-encrypted and chokes. Fix: re-encrypt it. Gotchas: (1) the bare docker exec ... bin/rails runner fails with key not found: ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY because the entrypoint-exported keys (from /rails/storage/.encryption_keys, persisted) don't reach an exec shell — use docker compose -p media-server run --rm --no-deps shelfarr bin/rails runner <file> which goes through the entrypoint. (2) Set ActiveRecord::Encryption.config.support_unencrypted_data = true to read the plaintext. (3) Assigning the SAME password value is a dirty-tracking no-op (won't re-encrypt) — toggle through a temp value: dc.update!(password:"tmp"); dc.update!(password:"Success402!"). (4) Restart the main container afterward so it reloads the row (Authenticated successfully to qBittorrent). qBit creds: user justinwieb, pw Success402!, url http://127.0.0.1:8080 (LocalHostAuth=false → localhost bypass).

  • Symptom (2): searches throw IndexerClients::Prowlarr ... Invalid Prowlarr API key (surfaced as a NameError: uninitialized constant AuthenticationError — shelfarr's buggy error handler). Cause: shelfarr's stored prowlarr_api_key (plaintext in settings table) drifted from Prowlarr's actual key (key rotation). Fix: update settings set value=<real key> where key='prowlarr_api_key' (real key: docker exec prowlarr grep -oE "<ApiKey>[^<]+" /config/config.xml), restart shelfarr. Prowlarr URL in shelfarr = http://localhost:9696/ (netns share).

  • Search behavior note: shelfarr searches the bare title with the apostrophe stripped (e.g. "Cat's Cradle" -> "Cats Cradle"), which can match junk; and it filters by the requested book_type (audiobook vs ebook). Many titles exist only as ebook (well-seeded EPUB) but not standalone audiobook. Request the format that actually exists; for Kindle reading, request ebook.

  • DURABLE FIX (2026-06-07) — hands-off ebook → Kindle. Root issue: shelfarr's ebook_output_path defaulted to /ebooks (= the calibre library root /mnt/storage/books), so PostProcessingJob copied raw files into the library dir where calibre/CWA never registered them (no metadata.db entry → invisible + no auto-send). Fix = repoint shelfarr's ebook output at the CWA ingest folder so importing is CWA's job: in shelfarr settings table set ebook_output_path=/downloads/books (shelfarr mounts /mnt/storage/downloads→/downloads; CWA ingest is /mnt/storage/downloads/books) and ebook_path_template='' (empty = flat files into the ingest, the proven-working pattern; CWA ingest recursion into subfolders is unconfirmed). Then chain is fully automatic: request → RequestQueueJob (every 5 min) → auto-select → DownloadJob → qBit → DownloadMonitorJob detects done → PostProcessingJob copies flat into ingest → CWA imports + kindle-epub-fixer + auto-send. Also set: auto_select_enabled=true, auto_select_confidence_threshold=70 (90 too strict, real scores ~80-86), auto_select_min_seeders=1; CWA per-user auto_send_enabled=1 (app.db user table). The earlier "download marked failed despite qBit 100%" symptoms were collateral from the qBit-password decryption bug (monitor couldn't auth to confirm) — gone once qBit auth was fixed. Lesson: RequestQueueJob runs on a 5-min cadence, so new requests aren't instant; don't assume "stuck" before waiting a cycle. See [[feedback_dont_handdrive_tests]].

PREFERRED ebook path (2026-06-07): manual search + qBit→CWA bridge

shelfarr's auto-search/auto-select proved flaky (wrong editions e.g. "Pride and Prejudice" → Zombies parody; grabbed PDFs; heavy automated searching rate-limited MAM). Justin's preferred flow = search manually, automate only the back half. shelfarr is now optional, not load-bearing.

  • Bridge (unified router): /opt/stacks/media/qbittorrent/forge_qbit_book_router.sh (= /config/... in the qbittorrent container). qBit autorun_enabled=true, autorun_program=/config/forge_qbit_book_router.sh "%F" "%N" (set via app/setPreferences; %F=content path, %N=name). Event-driven — qBit fires it ONCE per torrent at completion (not a watcher/cron), passing that torrent's path. Routes by extension found in %F:
    • *.epub/*.mobi → copy into CWA ingest /mnt/storage/downloads/books → CWA imports + converts + kindle-fixer + auto-sends to Kindle.
    • *.m4b/*.mp3/*.m4a/*.flac/*.ogg (+ cover/opf/nfo) → hardlink (preserve folder structure) into /mnt/storage/audiobooks/Justin Audiobook/<name>/ → Audiobookshelf. Hardlink = no extra disk, original keeps seeding (verified 2 links).
    • Both copy/hardlink leave the original in place → MAM keeps seeding (no Hit-and-Run). Logs to /config/forge_qbit_book_router.log. qBit mounts /mnt/storage→/mnt/storage 1:1 so paths match host; same filesystem (/dev/sda) so hardlinks work.
  • ABS auto-scan (zero-touch): after hardlinking an audiobook the router POSTs /api/libraries/c527db8b.../scan to ABS (localhost:13378, reachable since qBit+ABS share gluetun netns) using a Bearer token read from /config/.abs_token (= /opt/stacks/media/qbittorrent/.abs_token, chmod 600, ABS root user.token from absdatabase.sqlite). So new audiobooks appear in ABS with no manual scan. If the token ever rotates (ABS password change), re-stash it. Host DB: /opt/stacks/media/audiobookshelf/config/absdatabase.sqlite; libs Justin's=c527db8b..., Kristine=494cd59b..., Personal=bbafc550....
  • Proven 2026-06-07 with real MAM grabs: "Living an Examined Life" (epub→Kindle, auto-sent) + "A Life of Meaning" (m4b→ABS, 8.3h playable).
  • Justin's flow: find book on MAM (or Prowlarr search UI prowlarr.justinsforge.com) → download .torrent / grab → add to qBit (qbit.justinsforge.com, drag the .torrent) → walk away → lands on Kindle. EPUB/MOBI only (ebook_approved_formats/ebook_preferred_formats=["epub","mobi"] set in shelfarr too).
  • Optional upgrade pending: wire qBit as a Prowlarr download client so Prowlarr search → grab sends straight to qBit (one-click).