Eval Harness, Pure Phoenix Phase 4.5¶
Doctrine: FORGE-DOCTRINE.md Section 10.
Layout¶
| Path | Role |
|---|---|
forge/eval.json |
Check config (8 initial checks, severity ladder, comparators). |
forge/scripts/forge_eval_harness.py |
Runner. --staged for pre-commit, --full for nightly. |
forge/scripts/forge_eval_check_mkdocs_urls.sh |
Counts MD files in indexed dirs missing the URL line. |
forge/scripts/forge_eval_check_persona_code.sh |
Counts code-level retired persona names (Greg, OpenClaw, GregTwoPointO, Greg2). |
forge/scripts/forge_eval_check_service_names.sh |
Counts systemd units violating forge-<function> kebab-case (whitelist for deferred renames). |
forge/.git/hooks/pre-commit |
Calls harness in --staged mode. Blocks on fatal. |
forge/LESSONS.md |
Failure log. Harness appends one block per failed check per --full run. |
forge/logs/eval-harness.log |
Cron stdout/stderr. |
forge/logs/eval-harness-last-full.json |
Snapshot of last --full results. Used for regression detection. |
Cron¶
0 3 * * * /home/justinwieb/forge/scripts/forge_eval_harness.py --full --quiet >> /home/justinwieb/forge/logs/eval-harness.log 2>&1
Runs before 30 3 * * * search-index rebuild. On regression vs prior run, calls notify.sh warning ... (manager bot push).
Check set (initial 8)¶
| ID | Doctrine | Severity | What |
|---|---|---|---|
doctrine-present |
Sec 0 | fatal | forge/FORGE-DOCTRINE.md exists |
no-em-dashes |
Sec 9 | error | em-dashes in active scope <= 67 (LE; tighten as drift drops) |
memory-md-line-cap |
Sec 12 | error | MEMORY.md line count <= 200 |
memory-md-byte-cap |
Sec 12 | error | MEMORY.md bytes <= 25000 |
doctrine-mkdocs-serves |
Sec 9 | warning | mkdocs serves /FORGE-DOCTRINE/ with 200 |
mkdocs-url-on-new-md |
Sec 9 | warning | MD files in indexed dirs missing URL line |
no-persona-names-in-code |
Sec 3 | error | Greg/OpenClaw/etc in code files |
naming-taxonomy-services |
Sec 3 | warning | systemd units off forge-<function> pattern |
staged_relevant: true is the subset run pre-commit (5 of 8, drops the slow / runtime-dependent ones).
Severity ladder¶
| Level | Pre-commit | Nightly |
|---|---|---|
fatal |
blocks commit | LESSONS entry + manager bot page |
error |
prints, allows (Phase 4.5 initial; tightens to block after one clean week) | LESSONS entry + alert on regression |
warning |
prints, allows | LESSONS entry only, no alert |
Adding a new check¶
Edit forge/eval.json, append to checks[]. Required fields: id, doctrine_section, severity, command, expected, comparator. Optional: staged_relevant, description. Comparator must be one of equals, less_than_or_equal, greater_than_or_equal, contains. Run forge_eval_harness.py --full --no-write to verify before committing the new entry.
Pragmatic exceptions tracking¶
forge/LESSONS.md is the single source of truth for "this looks like a violation but is intentional, here is why." When a check would fail for a known-OK reason, either:
- Tighten the check's exclusion logic (preferred, makes it self-documenting in the script) or
- Log a LESSONS entry and bump the
expectedthreshold to absorb the known overage
Currently absorbed:
- 67 em-dashes (down to 55 actual on first run; mostly fenced-block ASCII art, doctrine literal references, retired-persona prompts inside lm-studio-setup archive)
- bot service names forge-telegram-ava.service + forge-telegram-inbox.service whitelisted; rename slated for Phase 4.2
[Claude Code, Pure Phoenix Phase 4.5]