feedback llm judge eval baseline parity 20260715T205329

During the CreatorTrack Gus voice-profile eval, a first run reported a 24/24 (100%) judge win rate for profiled-vs-baseline scripts, initially reported to Justin as a decisive PASS. A dedicated adversarial-audit agent, tasked specifically with attacking that PASS verdict, found the baseline system prompt had stripped not just the intended variable (voice card/exemplars) but also field-semantics guidance the treatment arm got. The judge's justifications showed it was identifying the baseline by format tells (bracketed camera directions, numbered steps) rather than voice, which also depressed the baseline's embedding similarity for unrelated reasons. The audit also caught selection bias in dropped/failed eval videos and overstated independence (6 of 20 holdout videos appeared in both of two 'independent' runs, so N was actually ~18 not 24).

Why: the assistant nearly shipped a false-positive PASS off a one-sided report; only a dedicated adversarial reviewer whose charter was to attack the eval's own methodology (not just re-check the numbers) surfaced the confound.

How to apply: when designing any LLM-judge or embedding-similarity A/B eval, isolate exactly one variable between arms; everything else (schema, formatting instructions, output structure) must be identical. When an eval returns a suspiciously clean result (unanimous win rate, too-good delta), route it through an adversarial reviewer charged with finding holdout leakage, unfair baseline setup, selection bias in failures, and overstated statistical independence, before reporting it to Justin as decided.

[auto-memory session 35bcc289-3fed-4eec-a68c-970364533914, confidence 0.75, mode staged]