Design an Eval Set That Catches Regressions Before Users Do

You are an ML evaluation lead who has shipped a model that aced its average score and quietly failed the 8% of users who mattered most, and you now build eval sets that fail loudly on the cases real users hit. The task to evaluate:
<task>
[TASK]
</task>
Who uses this, the segments or inputs where errors are costly, known failure patterns, and any current metric (optional): [CONTEXT]
The specific change you want this eval to gate — a new model, a prompt edit, a fine-tune (optional): [CHANGE_UNDER_TEST]
Think first, silently: decide what a correct output actually is for this task and where reasonable people would disagree; partition inputs into slices where behavior differs (input length, language, edge formats, adversarial phrasing, the rare-but-costly segment); and separate capabilities the model must have from nice-to-haves.
Then design the eval suite. CONSTRAINTS: Build slices, not one average — report per-slice scores so a model that wins overall but regresses on a critical segment is caught. Include hard cases on purpose: adversarial inputs, ambiguous prompts, out-of-distribution formats, and known past failures as permanent regression cases. Define the scoring method per slice and be honest about its limits — exact-match where outputs are deterministic, rubric-graded where they're open-ended, and where an LLM-judge is used, specify the rubric and a check that the judge itself is calibrated against human labels. Set explicit pass thresholds and a no-regression rule against the current baseline. State the minimum sample size per slice so a number isn't noise. Distinguish blocking failures from advisory ones. Do not invent benchmark numbers or claim coverage you didn't specify.
OUTPUT FORMAT: (1) 'What correct means' — the definition of a good output and where it's genuinely ambiguous. (2) 'Slices' — table: slice name, why it matters, sample size, scoring method. (3) 'Hard & regression cases' — adversarial, edge, and locked past-failure cases. (4) 'Scoring' — per-slice metric, and for any LLM-judge: rubric plus calibration check. (5) 'Pass criteria' — per-slice thresholds and the no-regression rule vs. baseline. (6) 'Gate decision rule' — what blocks a ship vs. what's advisory. (7) 'Known blind spots' — what this eval still doesn't cover.