Metric Diagnosis Workflow

Purpose

Systematically investigate why a metric changed unexpectedly (dropped or spiked) by segmenting data, distinguishing internal vs. external factors, and testing hypotheses to identify root cause. Prevents rushing to wrong conclusions and ensures evidence-based responses.

When to Use This Workflow

Use this workflow when:

• •Key metrics drop or spike unexpectedly
• •Leadership asks "what happened to [metric]?"
• •Post-mortem analysis needed after incident
• •A/B test shows unexpected patterns
• •Need to validate suspected root cause
• •User behavior changed suddenly without obvious reason

Skills Sequence

This workflow applies the root-cause-diagnosis skill three times, then assesses North Star impact:

1. Root Cause Diagnosis (Phase 1)
   ↓ (Segment across 4 dimensions: People, Geography, Technology, Time)
2. Root Cause Diagnosis (Phase 2)
   ↓ (Distinguish intrinsic vs extrinsic factors)
3. Root Cause Diagnosis (Phase 3)
   ↓ (Build hypothesis table and test against data)
4. North Star Alignment
   ↓ (Assess impact on top-line metrics)
   
OUTPUT: Narrowed scope, tested hypotheses, identified root cause, recommended actions

Required Inputs

Gather this information before starting:

Metric Change Information

• •The metric that changed
  • •Name and definition
  • •Example: "Weekly Active Users (WAU)"
• •Magnitude of change
  • •Absolute and percentage
  • •Example: "10,000 → 9,000 (10% drop)"
• •Timeframe
  • •When was it noticed?
  • •Over what period did change occur?
  • •Example: "Noticed today, change happened over past week"

Data Access

• •Segmented data availability
  • •Can you segment by user type, geography, platform?
  • •Are raw logs accessible?
• •Comparison data
  • •Historical baselines
  • •Year-over-year seasonality
  • •Competitor benchmarks (if available)

Recent Activity Log

• •Recent releases (past 1-4 weeks)
  • •Feature launches
  • •Bug fixes
  • •Infrastructure changes
• •Running experiments
  • •A/B tests active
  • •Configuration changes
• •External context
  • •Marketing campaigns
  • •News events
  • •Competitor activity

Workflow Steps

Phase 0: Data Quality Verification (5 minutes)

Before investigating, confirm the data is real:

1. •

   Check reporting systems

   • •Is tracking still functional?
   • •Any pipeline issues?
   • •Data freshness correct?
2. •

   Cross-reference sources

   • •Do multiple data sources show same pattern?
   • •Internal dashboards vs. external tools aligned?
3. •

   Validate sample sizes

   • •Sufficient data for statistical significance?
   • •Time periods comparable?

Red flags that indicate data quality issues:

• •Metric went to exactly zero
• •Change happened at exact midnight
• •Only visible in one data source
• •Coincides with known tracking deployment

If data quality issue found: Fix tracking first, then re-investigate if problem persists.

If data is valid: Proceed to Phase 1.

Phase 1: Narrow the Scope (15-20 minutes)

Use the root-cause-diagnosis skill - 4 Dimension Segmentation

Ask clarifying questions across all four dimensions:

Dimension 1: People (User Segments)

Questions to ask:

• •Does it affect all users equally?
• •Specific demographics (age, gender, location)?
• •New vs. returning users?
• •Free vs. paid users?
• •Power users vs. casual users?
• •Specific cohorts or user types?

Request data segmentation by user attributes.

Dimension 2: Geography

Questions to ask:

• •Specific countries or regions affected?
• •Certain cities?
• •Urban vs. rural?
• •Climate zones?
• •Time zones?

Request data segmentation by location.

Dimension 3: Technology (Platform)

Questions to ask:

• •iOS vs. Android?
• •Web vs. mobile app?
• •Desktop vs. mobile web?
• •Specific browser or OS versions?
• •Device types?
• •Network types (WiFi vs. cellular)?

Request data segmentation by platform/device.

Dimension 4: Time

Questions to ask:

• •When exactly did it start?
• •Gradual decline or sudden drop?
• •Day of week patterns?
• •Time of day patterns?
• •Recurring or one-time?
• •Seasonal effects?

Request time-series data with granular breakdown.

Output of Phase 1:

Narrow problem statement from:

• •❌ "Usage is down"

To:

• •✓ "iOS usage among teens in US dropped 10% starting Sept 1st"

Document:

## Narrowed Scope

**Original statement:** [Broad problem]
**Narrowed statement:** [Specific problem]

**Affected segments:**
- People: [Which user groups]
- Geography: [Which locations]
- Technology: [Which platforms]
- Time: [When and pattern]

**Unaffected segments:**
- [What stayed stable - equally important]

Phase 2: Generate Hypotheses (15-20 minutes)

Use the root-cause-diagnosis skill - Intrinsic vs. Extrinsic Factor Analysis

Brainstorm potential causes in both categories:

Intrinsic Factors (Internal Changes)

Engineering & Product:

• •Recent code releases (check deploy logs)
• •Bug introductions
• •Feature launches by any team
• •A/B experiments started/stopped
• •Algorithm or ranking changes
• •Performance degradations

Check with:

• •Engineering team (deploys, errors, performance)
• •Other product teams (their launches)
• •Data team (tracking changes)

Operations & Business:

• •Pricing changes
• •Policy updates
• •Support process changes
• •Marketing campaign changes

Check with:

• •Operations team
• •Marketing team
• •Customer success team

Extrinsic Factors (External Events)

Competition:

• •Competitor product launches
• •Competitor pricing changes
• •Competitor marketing campaigns

Market:

• •Economic shifts
• •Industry trends
• •Regulatory changes
• •Platform policy changes (iOS, Android, etc.)

Calendar/Seasonal:

• •Holidays
• •School schedules
• •Weather patterns
• •Cultural events

News/PR:

• •News coverage (positive or negative)
• •Social media trends
• •External events affecting user behavior

Check with:

• •Sales team (competitive intelligence)
• •Customer success (user feedback)
• •Marketing (market trends)
• •Industry news sources

Output of Phase 2:

List of 4-6 hypotheses (mix of intrinsic and extrinsic):

## Hypothesis List

**Intrinsic (Internal):**
1. [Hypothesis name]: [Description]
2. [Hypothesis name]: [Description]
3. [Hypothesis name]: [Description]

**Extrinsic (External):**
4. [Hypothesis name]: [Description]
5. [Hypothesis name]: [Description]
6. [Hypothesis name]: [Description]

Phase 3: Test Hypotheses (20-30 minutes)

Use the root-cause-diagnosis skill - Hypothesis Table Method

Step 3.1: Create Hypothesis Table

Columns: Data points observed (from Phase 1) Rows: Potential causes (from Phase 2)

Fill each cell with predicted impact:

• •↓ = Would decrease this segment
• •↑ = Would increase this segment
• •→ = No effect on this segment
• •? = Uncertain

Step 3.2: Match Patterns

For each hypothesis:

• •Do predictions match observations?
• •If ALL predictions match → Possible cause
• •If ANY predictions contradict → Rule out

Step 3.3: Request Differentiating Data

If multiple hypotheses fit:

• •Identify data points that would differ between them
• •Request additional segmentation
• •Narrow to single most likely cause

Example Table:

Pattern match: "School started (US)" fits all observations → Most likely cause

Output of Phase 3:

## Hypothesis Testing Results

**Ruled out:**
- [Hypothesis]: [Why ruled out]
- [Hypothesis]: [Why ruled out]

**Possible causes (remaining):**
1. [Most likely hypothesis]
   - Evidence supporting: [List]
   - Evidence against: [List if any]
   - Confidence: [High/Medium/Low]

2. [Second most likely] (if applicable)
   - Evidence supporting: [List]
   - Confidence: [High/Medium/Low]

**Additional data needed to confirm:**
- [Data point that would differentiate remaining hypotheses]

Phase 4: Assess North Star Impact (10 minutes)

Use the north-star-alignment skill

Evaluate whether identified root cause impacts company-level metrics:

Questions:

• •Does this affect North Star metrics?
• •Is impact temporary or permanent?
• •Does it require immediate action?
• •What's the business impact (revenue, growth, retention)?

Severity classification:

Critical (Act immediately):

• •Impacts revenue directly
• •Affects North Star metrics substantially
• •Permanent degradation if not fixed
• •Competitive disadvantage

Important (Act soon):

• •Impacts user experience
• •Affects intermediate metrics
• •May worsen over time
• •Needs monitoring

Low Priority (Monitor):

• •Expected variation (seasonality)
• •Affects non-critical segments
• •Self-resolving
• •No action needed

Output of Phase 4:

## North Star Impact Assessment

**Affected North Star metrics:**
- [Metric]: [Impact description]
- [Metric]: [Impact description]

**Business impact:**
- Revenue: [Direct/Indirect/None]
- Growth: [Slowed/Accelerated/Neutral]
- Retention: [Harmed/Improved/Neutral]

**Severity:** [Critical/Important/Low Priority]

**Urgency:** [Immediate/This week/Monitor]

Phase 5: Recommend Actions (10 minutes)

Based on root cause and severity, recommend next steps:

Action categories:

1. •

   Fix (for bugs/issues)

   • •What needs to be fixed?
   • •Who owns the fix?
   • •Timeline for fix?
2. •

   Mitigate (for external factors)

   • •What can reduce harm?
   • •Counter-measures available?
   • •Timeline for mitigation?
3. •

   Monitor (for expected variations)

   • •What tracking is needed?
   • •Alert thresholds?
   • •Review cadence?
4. •

   No Action (for acceptable changes)

   • •Why no action needed?
   • •What validates this decision?

Output document:

# Metric Diagnosis Report: [Metric Name]

## Executive Summary
[One paragraph: What happened, why, what to do]

## Problem Statement
- **Initial observation:** [Broad statement]
- **Narrowed scope:** [Specific statement with segments]
- **Magnitude:** [Absolute and % change]
- **Timeframe:** [When occurred]

## Root Cause Analysis

### Narrowed Scope (Phase 1)
- **Affected:** [User segments, geography, platforms, time pattern]
- **Unaffected:** [What stayed stable]

### Hypotheses Tested (Phase 2-3)
| Hypothesis | Category | Evidence | Conclusion |
|------------|----------|----------|------------|
| [Name] | Intrinsic/Extrinsic | [Supporting/Contradicting] | Ruled Out/Possible/Likely |

### Identified Root Cause
**Most likely cause:** [Hypothesis name]

**Evidence supporting:**
1. [Data point matching prediction]
2. [Data point matching prediction]
3. [Stakeholder confirmation]

**Confidence level:** [High/Medium/Low]

### North Star Impact (Phase 4)
- **Affected metrics:** [List with impact]
- **Severity:** [Critical/Important/Low]
- **Urgency:** [Immediate/Soon/Monitor]

## Recommended Actions

### Primary action:
**[Fix/Mitigate/Monitor/No Action]**
- What: [Specific action]
- Who: [Owner]
- When: [Timeline]
- Success criteria: [How to know it worked]

### Secondary actions:
- [Action 1]
- [Action 2]

### Monitoring plan:
- Metrics to track: [List]
- Review frequency: [Daily/Weekly]
- Alert thresholds: [When to escalate]

## Appendix
- Hypothesis table: [Detailed]
- Data sources: [Links]
- Stakeholders consulted: [Names/teams]

Common Diagnostic Patterns

Pattern 1: Seasonal Variation

• •Symptoms: Recurring pattern (weekly, yearly)
• •Investigation: Compare to historical same period
• •Action: Usually monitor, adjust baselines
• •Example: School year start, holiday season

Pattern 2: Competitor Impact

• •Symptoms: Sudden shift, specific demographic
• •Investigation: Check competitor news, market research
• •Action: Counter-strategy, feature parity
• •Example: Competing app launch with targeted marketing

Pattern 3: Internal Bug

• •Symptoms: Sudden drop, platform-specific
• •Investigation: Check recent releases, error logs
• •Action: Hotfix, rollback if needed
• •Example: iOS update broke key feature

Pattern 4: Product Change Side Effect

• •Symptoms: Gradual shift after release
• •Investigation: Analyze feature adoption patterns
• •Action: Iterate on feature, A/B test variations
• •Example: Redesign changed user behavior

Pattern 5: Tracking Issue (False Alarm)

• •Symptoms: Exact numbers, midnight timing, single source
• •Investigation: Verify data pipeline
• •Action: Fix tracking, dismiss false alarm
• •Example: Analytics tag removed in deploy

Common Mistakes

Success Criteria

Metric diagnosis succeeds when:

• •Data quality verified
• •Problem narrowed to specific segments (4 dimensions)
• •4-6 hypotheses brainstormed (intrinsic + extrinsic)
• •Hypotheses tested against data systematically
• •Root cause identified (or top 2-3 candidates with confidence levels)
• •North Star impact assessed
• •Recommended actions clear with owners and timelines
• •Diagnosis document created and shared
• •Stakeholders aligned on next steps

Real-World Example: Microsoft Teams Downloads Drop

Initial Report

"New app downloads down 50% from last week"

Phase 1: Narrow Scope (15 min)

Questions asked:

• •Geography? → "Predominantly US and Europe"
• •User type? → "Enterprise customers, not consumers"
• •Platform? → "Web and desktop primarily, some mobile"
• •Other metrics? → "Usage unchanged (existing users still active)"

Narrowed statement: "Enterprise customer downloads via web/desktop down 50% in US/Europe, starting last week. Existing user engagement unchanged."

Phase 2: Generate Hypotheses (15 min)

Intrinsic:

1. •Bug in download flow
2. •Recent release broke something
3. •A/B test reducing visibility

Extrinsic: 4. Marketing promotion ended 5. Competitor launch 6. Economic downturn affecting enterprise spending

Stakeholder consults:

• •Engineering: No recent releases, no bugs reported
• •Marketing: "We were running 50% off for enterprises switching from Slack"
• •Sales: No competitor news, pipeline healthy

Phase 3: Test Hypotheses (20 min)

Pattern match: "Promotion ended" fits all observations

Confirmation: Marketing confirms promotion ended last weekend

Phase 4: North Star Impact (5 min)

Root cause: End of promotional campaign

• •Downloads during promo = artificially inflated baseline
• •Current downloads = return to normal baseline

North Star impact:

• •MAU: No impact (existing users unaffected)
• •Revenue: No impact (existing customers retained)
• •Severity: Low (false alarm, not actual problem)

Phase 5: Recommended Actions (5 min)

Primary action: No action needed

• •Not an actual decline, just return to baseline
• •Adjust baselines for future comparisons

Secondary actions:

• •Update dashboards to flag promotional periods
• •Establish "normal" baseline for future monitoring
• •Document for future reference (prevent false alarms)

Time to diagnosis: 60 minutes

Related Skills

This workflow orchestrates these skills:

• •root-cause-diagnosis (Phases 1-3) - Core investigation method
• •north-star-alignment (Phase 4) - Assess strategic impact

Related Workflows

• •tradeoff-decision: May use diagnosis to understand why metrics changed before deciding
• •metrics-definition: Uses similar systematic approach for defining vs. diagnosing

Time Estimate

Total: 60-90 minutes

• •Phase 0 (Data quality): 5 min
• •Phase 1 (Narrow scope): 15-20 min
• •Phase 2 (Hypotheses): 15-20 min
• •Phase 3 (Test): 20-30 min
• •Phase 4 (Impact): 10 min
• •Phase 5 (Actions): 10 min

For complex issues, may require multiple iterations or longer investigation time.