Windtunnel method
Shortlist lift
Shortlist lift is the New message’s shortlist rate minus the Current message’s shortlist rate, in percentage points, reported with a run-seeded scenario-cluster bootstrap interval.
One difference, everything else held fixed.
A Message Lift test runs one Current message and one New message through shared contexts, so only the message changes. Shortlist lift is the difference between the two shortlist rates, in percentage points. Both rates come from the same AI model, the same scenarios, and the same disclosed prompts.
The interval is a bootstrap that resamples scenario clusters within the run, because scenarios in one run share a context and are not independent draws. A lift that crosses zero on both sides of that interval is reported as no lift, and a no is a finding.
Reading a lift.
The landing page’s Message Lift section shows the Current/New structure the lift is computed from. Public worked examples of Message Lift results, including one that said no, are linked from What we found.
Three things a shortlist lift is not.
- Not a sales forecast. It measures a change in simulated recommendations between two messages, never revenue, never a sales prediction.
- Not a head-to-head between brands. The two arms are two messages for one brand; competitor rates are context, not the result.
- Not a ranking prediction. A lift describes the tested model, prompts, and date. It does not promise where a model will place you next.
Labels travel with numbers: figures traced to stored answers from real model routes are MEASURED; Message Lift results are SIMULATED and comparative only; anything below the n≥30 gate or drawn from a single prompt cell is DIRECTIONAL. No figure on this site ships without its label, sample size, and date.