Pairwise comparison

How the pairwise comparison method works

Every item is matched against every other item once. With n items the full set of pairs is:

pairs = n × (n − 1) ÷ 2
4 items → 6 pairs; 6 items → 15 pairs; 10 items → 45 pairs

The method goes back to Louis Thurstone’s work on comparative judgement in the 1920s. It later became the basis of Thomas Saaty’s Analytic Hierarchy Process (AHP), where people also say how much more they prefer one option. A survey pairwise question usually asks only which one wins.

Pairwise comparison examples

• Which matters more to you: faster delivery or a lower price?
• Which of these two logo options fits the brand better?
• Given one extra week, which would you fix first: search or notifications?

The same question text repeats; only the pair changes. Keep that text short and neutral, because people read it once and then answer on reflex.

From choices to a ranking: win rate

win rate = times the item won ÷ times the item was shown × 100%

Sort items by win rate and you have a priority list. Unlike a ranking question, the gaps mean something: 80% against 45% is a real lead, 52% against 49% is a tie. Always check the number of comparisons per item; a win rate from three duels is noise.

When pairwise beats ranking, and when it does not

Use pairwise comparison for 6 to 12 items, when a ranking would have a random middle. It also works well for visual choices, where comparing two images side by side is natural.

Avoid it for very long lists answered by one person: 20 items produce 190 pairs, which nobody will finish. Show a subset of pairs to each respondent instead, and remember that each respondent then contributes to only part of the picture.

Pitfalls in pairwise studies

• Inconsistent choices (A over B, B over C, C over A). A few are normal; many mean the items are too similar or badly described.
• Items of very different size in one set (“new logo” against “free shipping”).
• Reading a 51% win rate as a preference when it rests on a handful of comparisons.

Building it in Askyo

The Pairwise comparison block takes at least two items and, by default, shows every possible pair. You can set how many pairs to show; the block then shows the first N pairs in a fixed order, not a random sample, so with a limit the same pairs go to everyone. Order the item list with that in mind.

All pairs sit on one screen as “A vs B” buttons, stacked on a phone and in three columns on wider screens, with a “Compared X of Y” counter. Analytics shows each item’s win rate (wins divided by comparisons) and how many comparisons it took part in. In the CSV each pair reads “A vs B → winner”.