Correlation vs Causation in Product Analytics

Short answer: two things moving together does not show that one causes the other. In product analytics the classic error is "users who do X retain better, so we should push everyone to do X". Often the users who do X were already more committed. To get closer to cause, look for a plausible mechanism in real sessions, check for a hidden third factor, and where possible run an experiment.

Table of Contents

1. The Famous Example
2. Four Ways Correlation Misleads
3. How to Test a Causal Claim
4. What Recordings Add
5. Phrasing Findings Honestly
6. Key Takeaways

The Famous Example

A team discovers that users who invite three teammates in their first week almost never churn. They redesign onboarding to push invitations. Invitations rise. Retention does not.

The users who invited teammates did so because they already saw value and worked in teams. The invitation was a symptom of commitment, not its cause. Forcing the behavior on uncommitted users changed nothing.

Four Ways Correlation Misleads

1. Reverse causation

You assume A causes B when B causes A. "Customers who contact support churn more" may mean that people who are already struggling contact support.

2. A hidden third factor

Something else drives both. Users on larger plans use more features and retain better. Company size drives both.

3. Selection

The group chose itself. People who opt into a beta, read the docs, or attend a webinar are different from those who did not, before the thing you are measuring happened.

4. Coincidence

Look at enough metrics and some will move together by chance. A release and a holiday in the same week will each be credited with whatever happened.

The last row is common enough to have its own name (Simpson's paradox).

How to Test a Causal Claim

1. Ask whether the order is right. Did A happen before B, for each user?
2. Look for a third factor. Split by plan, company size, acquisition source, and device. Does the relationship survive inside each group?
3. Compare like with like. Among users who were equally active on day one, does the behavior still predict retention?
4. Check the dose. If more of A goes with more of B in steps, the case is stronger.
5. Find a mechanism. Can you explain, concretely, how A would produce B?
6. Run an experiment. Randomly encourage half of new users to do A. If B improves for that half, you have real evidence.
7. Use natural experiments. A feature that was unavailable for a week, or rolled out to one region first.

An experiment is the only strong test. The others raise or lower your confidence.

What Recordings Add

Numbers can show that two things go together. Watching sessions can show whether there is a mechanism.

Take the invitation example. Watch ten users who invited teammates and ten who did not.

• If inviters had already built a project and used it before inviting, the invitation followed value. It was not the source of it.
• If non-inviters were solo users with no one to invite, the feature is irrelevant to them.
• If non-inviters looked for the invite option and could not find it, there may be a real, fixable cause.

Three different conclusions, indistinguishable in the chart. Identify users and track the behavior so the two groups can be filtered:

deepsync("track", "teammate_invited");

Recordings do not prove causation either. They tell you whether the story is plausible before you spend a quarter building on it.

Phrasing Findings Honestly

The honest version is less exciting in a meeting. It also stops the team from investing in the wrong thing.

Key Takeaways

• Behaviors that predict retention are often symptoms of commitment.
• Check order, third factors, and selection.
• Experiments are the strong test.
• Use recordings to check that a mechanism exists.

Conclusion

A correlation is a question, not an answer. Treat it as a lead to investigate, and you will build fewer features that change nothing.

Start free with DeepSync.