Skip to content

ERPNext / forge-biz-apps, State Review + Incident Recovery

URL: https://mkdocs.justinsforge.com/memory/handoffs/erpnext-biz-apps-state-review-2026-06-07/

Date: 2026-06-07 · Author: [Claude Code] (erp-review_Opus48) · Type: state review + live incident

TL;DR

  • ERPNext is the self-hosted, multi-company books engine on CT 104 (forge-biz-apps), public at erpnext.justinsforge.com behind Cloudflare Access. It replaced QuickBooks ($80/mo → $0/mo), decided 2026-05-03.
  • All 5 companies are seeded and confirmed live (WIEB parent + JWVR/NOVA/SIP subs + GOC equity-method), 494-account chart of accounts, FY2026.
  • It is a scaffold, not yet operational books: zero transactions of any kind (0 invoices, 0 journal entries, 0 bank accounts, 0 customers/suppliers/items). The Plaid bank-feed sync that justified the whole build was never wired.
  • Found a silent outage during this review and recovered it. Six core containers had been stopped since 2026-05-29 20:47 UTC (8–9 days); the site was returning 502s behind CF Access and nothing alerted. Brought the stack back up with the documented pwd.yml recovery. No data lost (266M MariaDB volume intact). Health is now 200 / pong, all 5 companies verified through the live API.

1. Architecture

Layer Detail
Host Finn (Proxmox), CT 104, hostname forge-biz-apps, Debian 12 unprivileged, nesting=on, 4c / 4 GB / 32 GB
IP 192.168.86.194 (DHCP; drifted from .66 ~2026-05-22). SSH alias biz-apps.
Stack Docker + frappe_docker, deployed via /opt/frappe_docker/pwd.yml (image frappe/erpnext:v16.16.0, project frappe_docker, net frappe_docker_frappe_network). Never use the default compose.yaml (unset env vars break the stack).
Services 9 long-running: backend, frontend (nginx), db (MariaDB), redis-cache, redis-queue, queue-long, queue-short, scheduler, websocket. Plus one-shots: configurator, create-site.
Public access https://erpnext.justinsforge.com via the media-server cloudflared tunnel (ingress → 192.168.86.194:8080), behind Cloudflare Access (allow [email protected] + [email protected]; app uid faefe1c2-b031-459c-8144-9e0f130e9f7e).
LAN/backend (bypasses CF Access) http://192.168.86.194:8080 with Host: erpnext.justinsforge.com header (site name is frontend).
Accounts Administrator (holds the API key Forge uses), [email protected] (System Manager / daily login), [email protected] not yet created.
Secrets ~/.forge-secrets/erpnext.env (admin key/secret, URLs, site). MariaDB root pw admin (container-internal only — never bind-mount/port-forward).

2. What is built and seeded

Seeded 2026-05-03 by scripts/forge_erpnext_seed_companies.py (idempotent). Verified live via API today:

Abbr Legal name Role
WIEB Wiebelhaus Enterprises LLC S-corp parent, Group company (is_group=1) for consolidation
JWVR JustinWiebVR LLC sub, parent = WIEB
NOVA Shop Nova Design LLC sub, parent = WIEB
SIP Sip 'N' Serve Society LLC sub, parent = WIEB
GOC Gus Outdoor Co standalone, 40% owned by WIEB via equity method
  • Chart of accounts: 494 accounts (Standard with Numbers COA × 5 companies), all USD / United States, calendar fiscal year. FY2026 only (FY2027 must be created before 2027-01-01).
  • Equity-method account 1850 - Investment in Gus Outdoor Co (40%) exists — but on all four non-GOC companies (WIEB, JWVR, NOVA, SIP), not WIEB-only as the reference file states. Only WIEB's is meaningful; the three on the subs are spurious.
  • Operational data: none. Journal Entry / Sales Invoice / Purchase Invoice / Payment Entry / Item / Customer / Supplier / Bank Account all = 0. The books are a structural skeleton; no bookkeeping has happened.

3. What replaced QuickBooks

  • Cost: QuickBooks ~$80/mo → $0/mo. ERPNext is the intended canonical source of truth for chart of accounts, P&L, balance sheet, invoicing, and multi-company consolidation across the Wiebelhaus stack.
  • Decision (2026-05-03): ERPNext over Odoo. Drivers: native Plaid + bank reconciliation (the "transactions land in a queue → click to categorize" QuickBooks workflow), free multi-company consolidation, MIT license (no open-core split), genuinely Claude-friendly REST API (every doctype is an endpoint).
  • Personal budgeting deliberately excluded (2026-05-04): ERPNext stays business-only; Firefly III was planned alongside on the same LXC for RocketMoney-style personal budgeting, fed by a shared Plaid bridge. Neither Firefly nor the bridge was built.

4. Incident 2026-06-07: silent outage found + recovered

Root cause: The 6 core containers (backend, db, redis-cache, redis-queue, queue-long, queue-short) cleanly exited (Exit=0, OOMKilled=false) at 2026-05-29 20:47 UTC and never came back. Their restart policy is on-failure, so a clean stop plus the later host reboot (~2026-06-03) correctly left them down. frontend + websocket (also on-failure, but exiting non-zero) crash-looped: nginx couldn't resolve backend:8000, websocket couldn't resolve redis-queue. The CF tunnel + Access layer stayed healthy, so the failure was masked as a 502 behind the auth gate.

Impact: ERPNext origin down ~8–9 days. No alert fired and no daily log records the 2026-05-29 stop — cause undetermined (not resource exhaustion: disk 19%, no OOM).

Recovery performed (read the live state, then fixed it): 

cd /opt/frappe_docker
docker compose -f pwd.yml up -d
# core services up, then kick the two that started before their deps had DNS:
docker restart frappe_docker-frontend-1 frappe_docker-websocket-1
Result: all 9 services Up, db healthy; local health /, /login, /api/method/ping → 200, ping body {"message":"pong"}; 5 companies returned by the live API. Data intact — stopping containers never touched the frappe_docker_db-data volume (266 MB).

5. Open threads / unfinished work

  1. No bank feed / no transactions. forge_plaid_sync.py (nightly Plaid → ERPNext) never built. Until it is, ERPNext holds no real books and QuickBooks isn't truly replaced in practice.
  2. forge_books custom Frappe app (Plaid bridge, equity-method posting cron, custom reports) — not built. The upgrade-safe layer for all customization.
  3. [email protected] service user not created; Forge still authenticates as Administrator.
  4. Firefly III (personal budgeting) — planned alongside, never deployed.
  5. No health monitoring. The daily canary covers spawn/bots/reaper but not ERPNext — which is why this outage was invisible for over a week.
  6. CT 104 has no static IP. DHCP drift (.66 → .194) already broke the tunnel once (05-22) and left ERPNEXT_URL stale (see below).
  7. ERPNEXT_URL in ~/.forge-secrets/erpnext.env still points at the dead 192.168.86.66 — any Forge automation using it fails with "no route to host." (The seed script worked today only because I overrode the base to .194.)
  8. Doc drift: reference files say IP .66 and "equity account on WIEB only"; reality is .194 and the account is on all 4 non-GOC companies.
  1. Add ERPNext to health monitoring — canary hit on http://192.168.86.194:8080/api/method/ping (Host header) or the public edge, push critical on failure. Highest value: this outage should never go silent again.
  2. Pin CT 104's IP — DHCP reservation or static config on Finn. Removes the recurring drift class (tunnel ingress + secrets both depend on it).
  3. Fix ERPNEXT_URL in ~/.forge-secrets/erpnext.env → http://192.168.86.194:8080 (do after #2 so it points at the pinned IP).
  4. Make the stack self-heal — a compose.override.yml setting core services to unless-stopped so a reboot/stop brings them back instead of crash-looping the frontend. (Don't edit upstream pwd.yml.)
  5. Build the Plaid bridge (forge_plaid_sync.py + forge_books) to make the books real — this is the actual point of the migration.
  6. Update the two reference memory files (IP, equity-account scope) to match verified reality.

Appendix: verification commands used

ssh biz-apps 'cd /opt/frappe_docker && docker compose -f pwd.yml ps'
ssh biz-apps 'curl -s -H "Host: erpnext.justinsforge.com" http://localhost:8080/api/method/ping'
# Companies (from Console; override base to current .194, UA must not be default Python):
#   GET http://192.168.86.194:8080/api/resource/Company  (Host + token + UA headers)