Why one AI answer is an anecdote.
Model answers are sampled from a distribution: ask the same question twice and the wording, the brands named, and the ranking can all differ. A single answer cannot tell you where you stand. Windtunnel treats the model itself as the measuring instrument and reads rates off repeated samples instead of anecdotes.
Every audit prompt runs five times.
Each audit prompt cell is sampled k=5 times per selected model route, so a rate like mention rate is computed over many stored answers, not one. A Message Lift test uses one AI model at k=5, because the comparison, not breadth, is the point there. Every stored answer is timestamped and traceable, so any figure can be re-derived from its source.
Repetition is also how honesty survives a bad day: a model that names you in two of five answers produces a very different rate from one that names you in four, and only repeated samples can tell those models apart. See repeated sampling for the term's definition.
Which numbers carry an interval.
Rate metrics with enough support carry a Wilson 95% confidence interval: mention rate and the shortlist-family rates. Some metrics are reported as point estimates on purpose: share of voice, average first position, and the stability index are computed in ways that break the assumptions an interval needs, and inventing one would be exactly the false precision this method exists to avoid.
Shortlist lift reports a bootstrap interval that resamples scenario clusters within the run, because recommendation scenarios in one run share a context and must not be treated as independent draws.
Small numbers get labels, not headlines.
Aggregate findings render only when at least 30 eligible samples support them. Below that gate, figures are labeled directional and never promoted into claims. Cell-level observations, such as a single lost-shortlist prompt, are always directional-only, no matter the sample size.
A result is a snapshot, not a law.
- Prompts. Conclusions hold for the tested prompt set, printed verbatim in every study.
- Routes. Every figure names the API/model route and mode that produced it. A consumer chat session can differ.
- Date. Models change without notice. A figure describes the day it was collected.
- Evidence class. Measured and simulated results are labeled and never mixed.
The method, one definition at a time.
Repeated sampling
Each prompt runs k=5 times per model and mode; rates describe the distribution.
Mention rate
Share of unbranded answers that name the brand unprompted.
Shortlist rate
Share of valid AI-recommendation answers that place the brand among the five ranked recommendations.
Top-choice rate
Share of valid AI-recommendation answers that rank the brand first.
Shortlist lift
New minus Current shortlist rate, in percentage points.
Stability index
Mean pairwise Jaccard of top-5 tracked-brand sets across repeated samples.
The response is not the shortlist.
Buyer response
A synthetic panel produces free-text reactions, not numeric ratings. Embedding similarity against versioned anchor statements maps reactions onto a five-point Likert probability distribution. Response lift compares point-estimate means on that construct. No invented confidence intervals, purchase probabilities, or sales predictions.
AI recommendation
Each message is supplied as untrusted context to the same brand-neutral shopping situations. Five ranked JSON recommendations are parsed deterministically. Top-five and top-choice rates describe the simulated recommendation outcome; Shortlist lift is the difference in percentage points, with scenario-cluster bootstrap intervals.
Aggregate results require at least 30 eligible samples. Persona slices remain directional. Both test types disclose exact A/B prompts and use shared contexts: only the message changes. Simulation rows never enter measured audit aggregates.
Inspect the published studies ↗From free-text answer to labeled number.
Nothing in the scoring is a black box. A stored answer becomes a distribution you can inspect and reproduce, and the anchor method comes from independent peer-reviewed research (arXiv:2510.08338), which Windtunnel productizes.
Stored answer
A free-text response, immutable and timestamped.
Embedding
The answer becomes a vector: a position in meaning-space.
Cosine vs anchors
Similarity against anchor statements: 5 intent levels × 6 wordings = 30 anchors.
Distribution
A probability mass across the 1 to 5 scale, not a single guess.
Labeled result
Response lift compares mean scores on the 1–5 construct, with n and its simulated label. No confidence interval is invented.
SimulatedAI recommendation follows a different path.
Each message is supplied as untrusted context to the same brand-neutral shopping situations. Exactly five ranked recommendations are parsed deterministically. Top-five and top-choice rates describe the result; Shortlist lift compares Current and New in percentage points, with scenario-cluster bootstrap intervals.
Inspect both scoring methods ↗Every Windtunnel figure ships in the Glass Box: labeled measured or simulated, gated by sample size, wrapped in its confidence interval where the math supports one, and traceable to the stored raw response behind it.
What every result carries.
- Repeated sampling. Audit prompts run multiple times because model answers vary. Rates describe that distribution.
- Measured versus simulated. Visibility evidence comes from stored model answers. Both Message Lift test types are comparative simulations. The two evidence classes never mix.
- Current/New prompt parity. Both messages use the same prompts and settings. Only the message changes.
- Provider and model disclosure. Every figure names the API/model route and mode that produced it.
- Sample size and uncertainty. Rates show n and confidence intervals where the metric supports one.
- Results that say no. A study that only ever confirms the brief is not measuring anything. Where the evidence does not support a claim, the report says so, and that is the finding.
- Snapshot limitations. Conclusions apply only to the tested prompts, routes, modes, and date.
See the method in a complete study.
The SK Jewellery investigation separates a visibility audit, an AI recommendation test and simulated buyer responses, with exact prompts and directional breakdowns.
Read the SK Jewellery study →