The Novelty Effect and Regression to the Mean — Why Early Results Fade

Short answer: the novelty effect is a temporary lift caused by something being new: returning users click a new button out of curiosity. Regression to the mean is the tendency of an unusually bad or good period to be followed by a more ordinary one, with or without your intervention. Both make a change look more effective than it is. Look at new users separately and wait several weeks before judging.

Table of Contents

1. The Novelty Effect
2. Regression to the Mean
3. How They Combine
4. How to Tell a Real Improvement
5. The Opposite Problem: Change Aversion
6. Key Takeaways

The Novelty Effect

You add a new feature to your dashboard, with a highlighted badge. Usage is high in week one. By week five it has settled at a fraction.

Existing users noticed something different and tried it. That is curiosity. It tells you the feature was discoverable, and nothing about whether it is useful.

The same happens with redesigned pages, new banners, and new email formats. Returning visitors respond to the change itself.

Who is not affected: new users. They never saw the old version, so nothing is novel to them. This is the key to measuring the true effect.

Regression to the Mean

Any metric fluctuates around its typical level. An unusually bad week is, more often than not, followed by a week closer to normal, simply because extremes are rare.

Now consider when teams act. Conversion has a terrible week. An urgent fix is shipped. The next week is better. The fix gets the credit.

It might deserve it. It might also be that the bad week was a random low and the recovery would have happened anyway.

This catches people constantly:

• The page that performed worst last month improves after you rewrite it.
• The campaign with the worst week does better after you change the creative.
• The salesperson with the worst quarter improves after coaching.

In each case, the worst performer was partly unlucky, and luck does not repeat.

How They Combine

The most misleading sequence is common:

1. A metric has a bad stretch (a random low).
2. The team ships a redesign.
3. The metric recovers (regression to the mean) and overshoots (novelty).
4. A success is announced.
5. Six weeks later the metric is back where it started, and nobody revisits the announcement.

How to Tell a Real Improvement

1. Look at new users only. No novelty effect applies. If new visitors convert better on the new version, the gain is more likely real.
2. Wait. Give a change four to six weeks before judging, longer for products used infrequently.
3. Watch the trend, not the first week. A lift that holds steady is real. One that decays toward the old level was novelty.
4. Use a control group. A proper A/B test sidesteps regression to the mean, because both groups experience the same random conditions (testing correctly).
5. Compare with a longer baseline. Measure against the average of the previous two or three months, not against the bad week that triggered the change.
6. Check a metric the change should not affect. If that rose as well, something else is going on.
7. Watch sessions. Is the specific friction you targeted gone? If the old problem still appears in recordings, the improvement in the chart is not from your fix.

For repeat usage, track whether people come back to the feature, not whether they tried it once:

deepsync("track", "report_exported");

Count users who did it in week four, not week one.

On Business plans, Pulse records the state of a metric when a finding is marked fixed and re-checks it on later runs, reporting a verdict of verified, improved, or inconclusive. A resolved problem that returns is reopened as a recurrence.

The Opposite Problem: Change Aversion

Novelty can also work against you. Returning users who knew the old layout are slowed down by a new one. Metrics dip, complaints arrive, and the temptation is to roll back.

Again, look at new users. If they perform better on the new design while existing users are temporarily worse, the dip is adjustment, and it usually passes within a few weeks.

Key Takeaways

• New things get tried because they are new.
• Bad weeks tend to be followed by better ones without any help.
• Judge changes on new users, over weeks, against a long baseline.
• A dip among existing users after a redesign may be temporary.

Conclusion

The first two weeks after a change are the least reliable weeks to measure it. Be patient, and separate the people for whom it was new from the people for whom it simply was.

Start free with DeepSync.