feedback gus voice profile eval unfair baseline pattern 20260715T210435

When building a baseline/control arm for an eval (isolating one variable, e.g. "voice profile present vs absent"), it's not enough to strip the feature under test from the prompt. Any other guidance the treatment arm gets (e.g. field-semantics, schema explanations) must be mirrored in the baseline verbatim, or the baseline underperforms for reasons unrelated to the tested variable, inflating the measured effect.

Why: In the CreatorTrack Gus voice-profile eval (feat/social-data-layer-and-scripts, Task 18), the baseline prompt omitted "text = actual spoken content not a label" and "shot_list = visual direction not spoken words" guidance that the profiled prompt had. The baseline arm then wrote camera-direction content into spoken fields, and the blind judge correctly flagged this as "doesn't sound like the creator" — but that was a schema-understanding artifact, not a voice signal. This inflated the win rate from 24/24 unfair to 17/17 fair (still a real pass, but the margin dropped: mean similarity delta +5.2pp -> +4.9pp, positive-delta rate 87.5% -> 70.6%).

How to apply: When designing or reviewing any eval that compares "feature on" vs "feature off" via an LLM judge, explicitly diff the two system prompts line by line and confirm every difference is attributable to the tested variable. Also watch for: correlated/overlapping samples across "independent" runs being double-counted as independent trials, and check whether failed/dropped samples correlate with the content type the feature is meant to help with (selection bias).

[auto-memory session 60d132de-4659-485c-8044-427af0ccce7f, confidence 0.72, mode staged]