feedback llm judge eval baseline parity 20260715T205316

During the CreatorTrack Gus voice-profile eval (Task 18), a first run reported a 24/24 (100%) judge win rate for profiled-vs-baseline scripts plus a consistent embedding-similarity delta, and this was initially reported to Justin as a decisive PASS. A second agent tasked with adversarially auditing that PASS verdict (specifically because a perfect score is exactly when methodology should be suspected) found the baseline system prompt had stripped not just the voice card/exemplars (the intended variable) but also the field-semantics guidance the treatment arm got (e.g. what belongs in beats.text vs shot_list). The judge's own justifications showed it was repeatedly identifying the baseline by format tells (bracketed camera directions, numbered steps) rather than voice, which also depressed the baseline's embedding similarity for reasons unrelated to voice. The audit also caught unaddressed selection bias in dropped/failed eval videos and overstated independence (6 of 20 holdout videos appeared in both of two 'independent' runs, so N was actually ~18 not 24).

Why: the assistant almost shipped a false-positive PASS to Justin off a one-sided report; only spawning a dedicated adversarial reviewer against the eval's own methodology (not just re-checking the numbers) surfaced the confound.

How to apply: when designing any LLM-judge or embedding-similarity A/B eval, isolate exactly one variable between arms, everything else (schema, formatting instructions, output structure) must be identical. When an eval returns a suspiciously clean result (unanimous win rate, too-good delta), route it through an adversarial reviewer whose charter is explicitly to attack the verdict (holdout leakage, unfair baseline, selection bias in failures, overstated statistical independence) before reporting it to Justin as decided. See [[reference_eval_harness]] for the separate doctrine-adherence eval harness (different system, same 'audit before trusting a green check' principle).

[auto-memory session 60d132de-4659-485c-8044-427af0ccce7f, confidence 0.75, mode staged]