Evaluating AI Output Without Fooling Yourself

Every team evaluates prompts. Most do it by pasting a request into a playground, reading the answer, nodding, and shipping. That process cannot detect a regression, and it quietly rewards prompts that happen to suit whatever example you last remembered.

Start with thirty real examples

Pull them from production logs, not imagination. Include the boring middle, the two known failure modes, and one input that should be refused. Thirty examples with honest labels beat three hundred synthetic ones.

Grade with the cheapest method that works

• Deterministic checks first: valid JSON, required fields, length limits, forbidden phrases.
• String or numeric match where a single correct answer exists.
• Model-as-judge only for open-ended quality, with a written rubric and a score scale.
• Human review on a sampled slice, to keep the judge honest.

A judge prompt needs the same rigour as the prompt it grades: fixed rubric, defined scale, and a spot check against human labels before you trust its numbers.

Report the numbers that decide things

prompt v4  vs  v3
pass rate      0.86  ->  0.91
schema valid   0.98  ->  1.00
p95 latency    2.4s  ->  3.1s
cost / call    $0.004 -> $0.011

Displayed like this, the tradeoff is a decision rather than a feeling. Sometimes you accept the latency; sometimes the five points are not worth tripling cost.

Run it on every change

Wire the harness into CI so any prompt, model, or retrieval change reports a diff. The value is not the score — it is that nobody has to argue from memory about whether last week's edit helped.

You do not need a platform. You need thirty examples, a grader, and the discipline to run it before merging.