Compare complete histories on matched cases.
Preregister a baseline and candidate, retain every attempt, normalize observed usage without filling gaps, and compare matched training and held-out cases.
How it works
How it actually works
Exactly what it does — and when.
A team freezes three training cases, one held-out case, and the rubric before running either prompt. Paired Experiments retains a failed candidate retry and an unavailable cost field, then reports the matched evidence without converting that gap into a savings claim.
This vs the alternative
“Freeze the experiment contract before seeing results.”
What you can do with it
Compare a prompt candidate
Preregister cases and a rubric before running the current and proposed instructions.
Evaluate a routing change
Keep all attempts and measured usage for matched accepted tasks.
Check held-out behavior
Test a separate case before deciding whether the candidate generalizes.
Inside the box
The shift it creates
- The metric is chosen after results
- Failed attempts disappear from totals
- Missing usage is treated as free
- Freeze the experiment contract before seeing results
- Retain retries, failures, and rework in each history
- Keep missing telemetry unknown and held-out cases separate
Who it's for
Agent evaluators
need paired evidence before adopting a change
Routing researchers
need complete task histories rather than cherry-picked calls
Teams measuring prompt changes
need training and held-out cases kept distinct
Questions, answered
Ready to put Genesis Paired Experiments to work?
Instant download · one-time price · own forever · all sales final