A/B Testing Skill
Purpose
Enables experimentation on lead generation sites to optimize conversion rates through simple, privacy-friendly client-side testing with statistical rigor.
Core Rules
- •One test at a time — Per page, avoid interaction effects
- •Statistical significance — Don't call winners too early (p < 0.05)
- •Track conversions — Not just clicks or exposures
- •Minimize flicker — Apply variants before paint using visibility hidden
- •Respect privacy — No PII in experiments, localStorage only
- •Minimum duration — Never less than 1 week (day-of-week effects)
- •Sample size first — Calculate required n before starting
- •Persist assignments — User sees same variant across sessions
- •Track exposures — Log every variant view to GA4
- •Conversion-focused — Optimize for business metrics, not vanity metrics
Test Duration Guidelines
| Traffic/Day | Min Duration | Sample Size |
|---|---|---|
| 100 | 8+ weeks | ~400 per variant |
| 500 | 2 weeks | ~400 per variant |
| 1000+ | 1 week | ~400 per variant |
Never call a test in less than 1 week — Day-of-week effects matter.
Common Tests for Lead Gen
| Test | Variants |
|---|---|
| CTA Text | "Get Quote" vs "Get Free Quote" vs "Start Now" |
| CTA Color | Primary vs Accent vs Contrasting |
| Form Length | 3 fields vs 5 fields |
| Social Proof | With reviews vs Without |
| Urgency | None vs "Limited slots" |
| Hero Image | Photo A vs Photo B |
| Headline | Benefit-focused vs Problem-focused |
GA4 Custom Dimensions
Set up in GA4 for tracking:
| Dimension | Scope | Description |
|---|---|---|
ab_test_id | Event | Test identifier |
ab_variant | Event | Variant name |
References
Detailed implementation guides and code examples:
- •Variant System — ABTest and ABVariant components, usage examples
- •GA4 Integration — Conversion tracking, GTM configuration, custom dimensions
- •Statistics — Sample size calculator, results analysis, significance testing
Forbidden
- •❌ Calling winners without significance (p < 0.05)
- •❌ Running less than 1 week
- •❌ Multiple tests on same element
- •❌ Changing tests mid-run
- •❌ Not tracking actual conversions
- •❌ Flicker (variant change visible to user)
- •❌ Storing PII in localStorage or events
- •❌ Skipping sample size calculation
Definition of Done
- • A/B test component implemented (ABTest.astro)
- • Variant content component implemented (ABVariant.astro)
- • Variant assignment persisted in localStorage
- • Exposures tracked in GA4 (ab_test_exposure event)
- • Conversions tracked in GA4 (ab_test_conversion event)
- • No visible flicker (visibility: hidden until variant applied)
- • Sample size calculated based on baseline conversion rate
- • Test runs minimum 1 week before analysis
- • GA4 custom dimensions configured (ab_test_id, ab_variant)
- • Statistical significance verified (p < 0.05) before declaring winner