Analytics Design

Purpose

Design a comprehensive measurement strategy including metrics hierarchy, event taxonomy, funnel instrumentation, and A/B test framework to ensure features can be evaluated with data.

Inputs

• •Feature or product area to instrument
• •Business goals and key questions to answer
• •Existing analytics infrastructure (tools, event patterns, naming conventions)
• •Privacy requirements and data retention policies
• •Target user segments for experimentation

Process

Step 1: Define Key Metrics Hierarchy

Build a three-tier metrics structure:

• •North Star Metric: The single metric that best captures the value delivered to users (e.g., weekly active users completing core action)
• •Leading Indicators: Metrics that predict movement in the North Star (e.g., activation rate, feature adoption, session frequency)
• •Guardrail Metrics: Metrics that must NOT degrade (e.g., page load time, error rate, unsubscribe rate, support ticket volume)

For each metric, define:

• •Precise definition (what counts, what doesn't)
• •Data source and calculation method
• •Current baseline (if known)
• •Target threshold

Step 2: Design Event Taxonomy

Establish a consistent naming convention:

• •Category: Feature area (e.g., editor, dashboard, onboarding)
• •Action: User action or system event (e.g., clicked, viewed, completed, errored)
• •Label: Specific element or variant (e.g., save_button, sidebar_nav, dark_mode)
• •Value: Numeric value if applicable (e.g., duration, count, score)

Naming rules:

• •Use snake_case for all event names
• •Format: category.action.label (e.g., editor.clicked.save_button)
• •Keep consistent across platforms (web, mobile, API)
• •Version events when schema changes (v2.editor.clicked.save_button)

Step 3: Specify Event Properties

For each trackable interaction, define:

Include:

• •Common properties (present on every event)
• •Feature-specific properties
• •Context properties (page, referrer, device, viewport)

Step 4: Plan Funnel Instrumentation

For each key user flow:

• •Define funnel stages (each stage = a specific event)
• •Identify drop-off measurement points between stages
• •Design recovery tracking (users who drop off then return)
• •Plan cohort segmentation (new vs returning, plan type, entry source)

[Stage 1: View] → [Stage 2: Engage] → [Stage 3: Convert] → [Stage 4: Retain]
     100%              65%                  30%                  20%
         ↓ 35% drop        ↓ 35% drop          ↓ 10% drop
    [Exit survey?]    [Tooltip help?]      [Follow-up email?]

Step 5: Design A/B Test Framework

For each planned experiment:

• •Experiment name: Descriptive, following convention (e.g., onboarding_v2_simplified_flow)
• •Hypothesis: "If we [change], then [metric] will [improve/decrease] because [reason]"
• •Variants: Control (A) and treatment(s) (B, C...)
• •Allocation: Percentage split per variant (typically 50/50 for two variants)
• •Sample size: Calculate minimum detectable effect (MDE) with 80% power, 95% significance
• •Duration: Estimated time to reach sample size based on traffic
• •Guardrails: Metrics to monitor for negative effects during the test

Step 6: Define Success Criteria

For each experiment:

• •Primary metric: The one metric that determines success
• •Minimum detectable effect: The smallest improvement worth shipping (e.g., +5% conversion)
• •Statistical significance threshold: Typically p < 0.05
• •Practical significance: Is the effect size meaningful even if statistically significant?
• •Decision framework:
  • •Significant positive result → Ship to 100%
  • •Significant negative result → Revert and analyze
  • •No significant result → Extend test or abandon
  • •Mixed results (primary up, guardrail down) → Investigate and decide

Step 7: Plan Data Pipeline and Storage

• •Collection method: Client-side SDK, server-side events, or hybrid
• •Transport: Real-time streaming vs batch upload
• •Storage: Analytics warehouse, time-series database, or SaaS platform
• •Retention: How long to keep raw events vs aggregated data
• •Access: Who can query the data, dashboards to create
• •Privacy: PII handling, consent tracking, GDPR/CCPA compliance

Output Format

Metrics Hierarchy

North Star: [Metric Name]
  ├── Leading: [Indicator 1]
  ├── Leading: [Indicator 2]
  ├── Leading: [Indicator 3]
  ├── Guardrail: [Metric A] (must not decrease)
  └── Guardrail: [Metric B] (must stay below threshold)

Event Catalog

Funnel Diagram

[Stage 1] ──→ [Stage 2] ──→ [Stage 3] ──→ [Stage 4]
  100%           ___%           ___%           ___%

A/B Test Plan

Quality Checks

• • North star metric is clearly defined with calculation method
• • Event taxonomy follows consistent naming convention
• • All key user interactions have corresponding events
• • Funnels have defined stages with drop-off measurement
• • A/B tests have hypotheses, sample size calculations, and duration estimates
• • Success criteria include both statistical and practical significance
• • Privacy and data retention requirements addressed
• • Guardrail metrics defined for each experiment

Evolution Notes