Why Your A/B Test Results Are Probably Wrong (and How to Fix Them)

Short answer: an A/B test only tells the truth if you decide the sample size in advance, run it for full weeks, judge it on one primary metric, and do not stop it the moment it looks good. Break any of those rules and a "winning" variant is often just noise. If you do not have the traffic to follow them, fix observed problems directly and measure before and after.

Table of Contents

1. Why False Winners Are So Common
2. Six Mistakes
3. A Checklist for a Trustworthy Test
4. When You Cannot Test
5. Using Recordings to Read a Result
6. Key Takeaways

Why False Winners Are So Common

Conversion is a noisy number. Two identical pages shown to two random groups will produce different conversion rates, purely by chance. A test's job is to tell a real difference from that noise. Every shortcut makes noise look like a result.

A useful exercise is an A/A test: run the same page against itself. If your process declares a winner, your process is the problem.

Six Mistakes

1. Peeking and stopping early

Checking the result daily and stopping when it reaches significance is the most common error. Results swing above and below the threshold as data accumulates. If you stop at the first crossing, you will declare winners far more often than the stated error rate suggests.

Fix: set the sample size before you start and do not act until you reach it.

2. Too little data

With a few dozen conversions per variant, a handful of people decide the outcome.

Fix: as a rough working rule, you want hundreds of conversions per variant before trusting a modest difference. Use a sample size calculator with your real baseline rate and the smallest improvement you care about.

3. Not running full weeks

Weekday and weekend visitors behave differently. A test from Monday to Thursday measures a different audience from one that includes a weekend.

Fix: run in whole weeks, at least two.

4. Many metrics, one winner

Track twenty metrics and one will show a "significant" change by chance.

Fix: name one primary metric before starting. Treat the rest as context.

5. Slicing after the fact

The test showed nothing overall, so you look by device, country, and source until a segment "wins". This is the same problem as too many metrics.

Fix: decide segments in advance, or treat a segment finding as a hypothesis for a new test.

6. Broken assignment

Variants receive different kinds of traffic, visitors see both versions, or one variant loads more slowly and loses people before it renders.

Fix: check that the split matches what you configured. A 50/50 test showing 55/45 has a fault. Check errors and load time per variant.

A Checklist for a Trustworthy Test

• [ ] A written hypothesis with the evidence behind it
• [ ] One primary metric
• [ ] Sample size calculated in advance
• [ ] Duration in full weeks
• [ ] Traffic split verified
• [ ] No stopping early
• [ ] No other major changes during the test
• [ ] Result recorded, including tests that showed nothing

When You Cannot Test

Many sites do not have the volume. With two hundred conversions a month, detecting a ten percent relative improvement can take most of a year.

That does not mean you cannot improve.

1. Fix what is plainly broken. A button hidden on mobile does not need a test.
2. Make bigger changes. Large effects need less data to detect.
3. Measure before and after, over matching periods, for the affected segment.
4. Use qualitative evidence. Watch sessions before and after the change and see whether the friction is gone.
5. Test higher in the funnel, where there are more events, such as clicks to the next step.

More in analytics for low-traffic websites.

Using Recordings to Read a Result

A test tells you which variant won. It does not tell you why, and it does not tell you why a test lost.

• Before the test: recordings supply the hypothesis (test ideas from session replay).
• During: watch a few sessions per variant to confirm each renders and works correctly on mobile.
• After: watch sessions from the losing variant. Often the idea was sound and the execution had a flaw.

Send the variant as a custom event so sessions can be found by it:

deepsync("track", "experiment_viewed", { experiment: "pricing-layout", variant: "b" });

Key Takeaways

• Conversion is noisy; shortcuts turn noise into "winners".
• Fix sample size and metric before you start, and run full weeks.
• Do not slice results until something wins.
• Low-traffic sites should fix and measure instead of testing.

Conclusion

A test is worth running only if you would believe it whichever way it came out. Set the rules first, and keep them.

Start free with DeepSync.