A/B Testing Results Explained: Lift, p-Values, and the Peeking Trap

Most A/B write-ups stop at "p less than 0.05, ship it." That is how teams ship noise, kill winning variants too early, and argue about lift percentages that nobody can reproduce.

This post does the opposite. You will download a real (synthetic but deterministic) checkout experiment, run the same SQL and Python we used, and read the results the way a careful analyst should: absolute and relative lift, a 95% confidence interval, a p-value with a plain-English meaning, and a peeking mistake that almost buried a clear win.

Download the data: experiment.csv (9,600 users, ~240 KB). Every number below comes from that file.

#What we are measuring

An online store tested a simplified checkout page for 14 days.

• Control: existing checkout
• Treatment: shorter form + clearer shipping total
• Primary metric: conversion (converted = 1 if the user completed an order)
• Secondary: revenue among converters (not the decision metric)

Each row is one user:

#Load the data and check balance

import pandas as pd
from math import sqrt
from statistics import NormalDist

df = pd.read_csv("experiment.csv")
print(df.groupby("variant").agg(
    n=("user_id", "count"),
    conversions=("converted", "sum"),
    rate=("converted", "mean"),
))

You should see roughly balanced arms (ours landed at 4,836 control and 4,764 treatment). Large imbalance is a randomization smell — pause before you celebrate any lift.

The same check in SQL (DuckDB, BigQuery, or Postgres):

SELECT
  variant,
  COUNT(*) AS n,
  SUM(converted) AS conversions,
  AVG(converted) AS conversion_rate
FROM experiment
GROUP BY 1
ORDER BY 1;

#The results table (full 14 days)

Absolute difference: +2.73 percentage points.
Relative lift: +31.3% (= 2.73 / 8.73).

#The two-proportion z-test, line by line

For conversion rates pc and pt with sample sizes nc and nt:

SE = sqrt( pc*(1-pc)/nc + pt*(1-pt)/nt )
z  = (pt - pc) / SE

c = df[df.variant == "control"]["converted"]
t = df[df.variant == "treatment"]["converted"]
pc, pt = c.mean(), t.mean()
nc, nt = len(c), len(t)

se = sqrt(pc * (1 - pc) / nc + pt * (1 - pt) / nt)
z = (pt - pc) / se
p_value = 2 * (1 - NormalDist().cdf(abs(z)))
ci_low, ci_high = (pt - pc) - 1.96 * se, (pt - pc) + 1.96 * se

print(f"z={z:.2f}  p={p_value:.6f}")
print(f"95% CI on difference: [{ci_low*100:.2f} pp, {ci_high*100:.2f} pp]")

On this file that prints:

• z ≈ 4.45
• p ≈ 0.000009
• 95% CI on the absolute difference: [+1.53 pp, +3.94 pp]

Plain English: if the two pages converted the same, a gap this large would be extremely rare. The interval also stays entirely above zero, so the result is not a fragile p-value that flips when you squint.

#What the p-value is — and is not

The p-value answers one question: assuming the null (equal rates), how surprising is this data?

It does not answer:

• the probability the treatment is better
• the probability you will see the same lift next month
• whether the change is worth the engineering cost

For size and uncertainty, read the confidence interval. For decision making, bring a minimum detectable effect you agreed on before launch — for example "we only ship if the lower CI bound is above +1 pp."

#The peeking trap (day 2 almost killed a winner)

Teams love to "just take a quick look." Here is what that look would have shown on day 2 (first ~21% of traffic):

On day 2, at α = 0.05, an impatient analyst calls the test a failure and rolls back. The same experiment, run to the planned sample, is a clear win.

early = df[df["day"] <= 2]
# Re-run the same z-test on `early` — you should get p ≈ 0.057

If you need interim looks, use a sequential testing design (or pre-declared peek schedule with alpha spending). Do not reuse a fixed-horizon z-test as a daily stop/go button.

#A practical decision checklist

1. Pre-register the primary metric, α, power, and MDE before traffic starts.
2. Check balance and data quality before any lift math.
3. Report absolute + relative lift and the 95% CI.
4. Treat p-value as evidence against the null, not a ship button.
5. Do not peek with a fixed-horizon test.
6. Separate primary (conversion) from guardrails (revenue per user, latency, refund rate).

Want more practice reading noisy product data? Work through a business case study or drill practical statistics on DSBootcamp — same habit of verifying numbers from the raw table.

#Where to go next

Re-run the notebook on the CSV above, then deliberately cut the file to day 2 and day 6 so you feel the peeking mistake yourself. When you are ready to turn experiment literacy into interview signal, pair this with hypothesis-testing problems on the practice tracks.