Simpson's Paradox in Analytics — When the Average Points the Wrong Way

Short answer: Simpson's paradox is when a trend seen in every subgroup reverses once the groups are combined. In analytics it usually happens because the mix of traffic changed. Mobile and desktop conversion can both rise while the overall rate falls, simply because a larger share of visitors is now on mobile, which converts lower. Always check segments before believing a blended number.

Table of Contents

1. A Worked Example
2. Why It Happens
3. Where It Shows Up
4. How to Protect Yourself
5. Which Number Is Right?
6. Key Takeaways

A Worked Example

A site redesigns its checkout. Illustrative figures:

Before

After

Desktop improved. Mobile improved. The total fell from 4.4% to 3.5%.

The redesign worked. What changed was a social campaign that tripled mobile traffic. A team looking only at the headline would roll back a successful change.

Why It Happens

An overall rate is a weighted average of segment rates. The weights are the share of traffic in each segment. If the weights shift toward a lower-converting segment, the average falls even when every segment gets better.

Two conditions produce the paradox:

1. Segments have very different rates.
2. The mix between them changes.

Both are normal on any website.

Where It Shows Up

• Device. The most common case. Mobile share grows and blended conversion falls.
• New and returning visitors. A campaign brings many new visitors, who convert less.
• Traffic source. Paid social scales up; overall rate drops.
• Geography. Expansion into a market with lower conversion.
• A/B tests. If variants receive different mixes of traffic, the comparison is distorted.
• Plan or customer type. Retention "falls" because you signed many small customers.
• Page type. Average time on site changes because blog traffic grew.

How to Protect Yourself

1. Segment before you conclude. Device, new versus returning, and channel at minimum.
2. Report the mix beside the rate. "Conversion 3.5%, mobile share 75%, up from 40%."
3. Compare segment to segment. This month's mobile against last month's mobile.
4. Hold the mix constant. Recalculate the new period using the old weights. In the example, the new segment rates with the old 60/40 split give 4.9%, an improvement.
5. Annotate campaigns and other events that change traffic composition.
6. Check test assignment. Each variant should have the same device and source mix.

In practice this is a filter away. Funnels, heatmaps, and session lists in DeepSync can each be split by device, source, and country, and compare mode puts two segments side by side (heatmap segments).

Which Number Is Right?

Both are correct. They answer different questions.

• The segment rates tell you whether the experience improved. Use them to judge a product change.
• The overall rate tells you what the business got from its total traffic. Use it for revenue planning.

The paradox is only a trap when you use the overall rate to answer the first question.

A caution in the other direction: segmenting is not a licence to keep slicing until you find the story you prefer. Choose the segments that plausibly matter before you look.

Key Takeaways

• A blended rate depends on the traffic mix as much as on performance.
• Every segment can improve while the total falls.
• Check segments and report the mix with the rate.
• Recalculate with constant weights to isolate real change.

Conclusion

An average hides its own composition. Before reacting to a number, look at what it is made of.

Start free with DeepSync.