A/B Testing: Is the Collections Treatment Strategy Working?

The Scenario

Pinnacle Finance is a consumer lending company managing a portfolio of over 50,000 overdue personal loan accounts. The Collections Strategy team recently developed a redesigned outreach script — referred to internally as the New Treatment — that emphasises repayment flexibility and personalised payment plans rather than the traditional firm-tone demand approach. Before rolling it out company-wide, the Head of Collections wants statistical proof that the new script actually improves debt recovery. A controlled experiment was run over 60 days: 800 overdue borrowers were randomly split into two groups — 400 received the standard script (Control) and 400 received the new script (Treatment) — and whether each borrower made a repayment within 30 days was recorded.

The Statistical Challenge

The collections team observed that 38% of borrowers in the Treatment group made a repayment, compared to 30% in the Control group. This looks like an improvement — but is the 8 percentage point difference real, or could it simply reflect random variation between the two groups? You have been brought in to conduct a formal A/B test using a two-proportion z-test, which compares repayment rates between the two groups and determines whether the observed difference is statistically significant. A core part of the analysis is also verifying that the experiment was adequately powered — i.e., that 400 borrowers per group is large enough to reliably detect a meaningful difference if one truly exists.

What's at Stake

The new collections script costs more to deliver — agents require additional training and calls take longer on average. If the uplift in repayment rate is statistically significant, Pinnacle Finance will invest in full-scale training and script adoption across its 200-agent collections floor, directly improving debt recovery revenue. If the difference is not statistically significant, the company avoids a costly rollout of a strategy that may not perform better than what they already have. Your statistical conclusion is the business trigger for a decision worth hundreds of thousands of dollars in operational investment.