The Long-Horizon Holdout Protocol
Overview
A measurement framework that challenges short-term growth wins by maintaining long-term control groups (holdouts) to verify if immediate lifts translate to sustainable value over 1-3 years.
Core principle: 30-40% of short-term "wins" show neutral or zero impact long-term.
The Process
code
┌─────────────────────────────────────────────────────────────────┐ │ T-0 (Launch) │ │ Split Traffic: 90% Treatment / 10% Long-term Holdout │ ├─────────────────────────────────────────────────────────────────┤ │ T+3 Weeks │ │ Initial Decision: Ship if short-term positive/neutral │ │ Keep holdout running │ ├─────────────────────────────────────────────────────────────────┤ │ T+6 Months │ │ Automated Ping: Check retention and emerging GMV patterns │ ├─────────────────────────────────────────────────────────────────┤ │ T+12-18 Months │ │ Final Reckoning: Compare Cohort GMV │ │ If neutral/negative: deprecate feature or pivot strategy │ └─────────────────────────────────────────────────────────────────┘
Key Principles
| Principle | Description |
|---|---|
| Long holdout | Keep control group for 12-18 months |
| Automated pings | System reminds team to check at 6, 12 months |
| Cohort GMV focus | Long-term value, not short-term conversion |
| Accept reversals | Be willing to deprecate "successful" features |
Common Mistakes
- •Declaring victory after 2 weeks of significance
- •Assuming neutral short-term = failure (might compound later)
- •Not setting up infrastructure for long-term tracking
Source: Archie Abrams (Shopify VP Product & Growth) via Lenny's Podcast