Tradeoff Decision Workflow

Purpose

Make informed ship/no-ship/iterate decisions when A/B test results are mixed (some metrics up, others down) or when a launched feature shows conflicting performance. Avoids binary thinking by exploring nuanced approaches that preserve gains while addressing losses.

When to Use This Workflow

Use this workflow when:

• •A/B test shows some metrics increasing, others decreasing
• •Feature launched with mixed performance across different metrics
• •Stakeholders disagree on whether to ship or rollback
• •Redesign shifted user behavior in unexpected ways
• •Need to choose between two good but mutually exclusive options
• •Optimizing for one metric clearly harms another

Skills Sequence

This workflow applies four skills systematically:

1. North Star Alignment
   ↓ (Determine which metric matters more strategically)
2. Trade-off Evaluation (Short vs. Long-term)
   ↓ (Assess temporal trade-offs)
3. Trade-off Evaluation (Mitigation)
   ↓ (Explore "best of both worlds" approaches)
4. Funnel-Based Metric Mapping
   ↓ (Identify where in user journey problem occurs)
   
OUTPUT: Assessment (net positive/negative/depends), 
        Recommended action (ship/rollback/iterate),
        Mitigation strategies, Risks

Required Inputs

Gather this information before starting:

Metric Changes

• •

  Metrics that increased

  • •Which metrics went up?
  • •By how much (absolute and %)?
  • •Statistical significance?
• •

  Metrics that decreased

  • •Which metrics went down?
  • •By how much (absolute and %)?
  • •Statistical significance?

Context

• •

  Change description

  • •What was tested or launched?
  • •What was the hypothesis?
  • •How long has it been running?
• •

  Company strategic priorities

  • •What's most important to company now?
  • •Growth, monetization, retention, competitive positioning?
• •

  User journey context

  • •Where in the funnel does change occur?
  • •Which user behaviors shifted?

Current State

• •

  Rollback feasibility

  • •Can we easily revert?
  • •What's the cost of reverting?
  • •How many users affected?
• •

  Iteration options

  • •Can we modify the change?
  • •What variations could we test?
  • •Timeline for iteration?

Workflow Steps

Step 1: Strategic Alignment Assessment (15 minutes)

Use the north-star-alignment skill

Determine which metric matters more to company strategy:

Questions to answer:

1. •

   Which metrics align with North Star?

   • •Do positive metrics connect to company-level goals?
   • •Do negative metrics impact North Star metrics?
2. •

   Which metrics align with mission?

   • •Does change serve company mission?
   • •Does it strengthen core value proposition?
3. •

   Which metrics drive long-term competitive advantage?

   • •Does change improve differentiation?
   • •Does it strengthen market position?
4. •

   Which metrics feed important flywheels?

   • •Do positive metrics create virtuous cycles?
   • •Do negative metrics break retention loops?

Output:

## Strategic Alignment Assessment

**Metrics Aligned with Strategy:**
- [Metric that went up/down]: [Why strategically important]
- [Metric that went up/down]: [Why strategically important]

**Strategic Priority Ranking:**
1. [Most important metric]: [Rationale]
2. [Second priority]: [Rationale]
3. [Lower priority]: [Rationale]

**Mission Alignment:**
- Change serves mission: [Yes/No/Partially]
- Rationale: [Explanation]

**Competitive Positioning:**
- Strengthens positioning: [Yes/No/Mixed]
- Rationale: [Explanation]

Step 2: Temporal Trade-off Analysis (15 minutes)

Use the tradeoff-evaluation skill - Short vs. Long-term Framework

Evaluate whether this is short-term gain/long-term loss or vice versa:

Framework questions:

1. •

   Short-term Impact (this quarter)

   • •Immediate revenue effect?
   • •This quarter's growth?
   • •Current user satisfaction?
2. •

   Long-term Impact (12+ months)

   • •User retention trajectory?
   • •Flywheel effects?
   • •Sustainable competitive advantage?
   • •Brand and market position?
3. •

   Irreversibility Assessment

   • •Can we reverse course later if needed?
   • •What becomes locked in?
   • •What habits are we building?

Apply Snapchat test:

• •Does this improve vanity metrics (time-on-site) while breaking engagement flywheel (messages/stories)?
• •Are we optimizing for wrong thing?

Output:

## Temporal Analysis

**Short-term (0-3 months):**
- Revenue impact: [Positive/Negative/Neutral - magnitude]
- Growth impact: [Positive/Negative/Neutral - magnitude]
- User satisfaction: [Positive/Negative/Neutral]

**Long-term (12+ months):**
- Retention impact: [Projected effect]
- Flywheel effects: [Strengthened/Weakened/Neutral]
- Competitive position: [Stronger/Weaker/Same]
- Strategic alignment: [High/Medium/Low]

**Trade-off Classification:**
- [Short-term gain, long-term loss] → Risky
- [Short-term loss, long-term gain] → Consider if affordable
- [Short-term gain, long-term gain] → Ship
- [Short-term loss, long-term loss] → Don't ship

**Recommendation based on temporal analysis:**
[Ship/Don't ship/Depends on mitigation]

Step 3: Mitigation Strategy Exploration (20-30 minutes)

Use the tradeoff-evaluation skill - Mitigation Framework

Before deciding to rollback or ship as-is, explore "best of both worlds" options:

Mitigation strategy categories:

Strategy 1: Segmented Rollout

When to use: Impact varies by user segment

Analysis:

• •Which segments show net positive?
• •Which segments show net negative?
• •Can we ship to positive segments only?

Implementation:

• •Define segment criteria
• •Rollout plan for positive segments
• •Iteration plan for negative segments

Strategy 2: Feature Modification

When to use: Specific element causing harm

Analysis:

• •Which specific change caused negative impact?
• •Can we keep beneficial parts, modify harmful parts?
• •What variations could preserve gains while fixing losses?

Implementation:

• •Identify problematic component
• •Design modified version
• •A/B test: Original vs. Modified vs. Current

Strategy 3: Compensation Mechanisms

When to use: Trade-off is necessary but painful

Analysis:

• •What complementary features could compensate?
• •Can we create new paths to value?
• •Can we boost other areas to offset loss?

Implementation:

• •Identify compensation opportunities
• •Parallel feature development
• •Monitor net effect

Strategy 4: Gradual Rollout with Monitoring

When to use: Uncertain about long-term effects

Analysis:

• •What could we learn from gradual rollout?
• •What delayed effects might emerge?
• •What monitoring would de-risk?

Implementation:

• •10% → 25% → 50% over weeks
• •Maintain holdback group
• •Set rollback trigger criteria

Strategy 5: Threshold-Based Approach

When to use: Acceptable within certain bounds

Analysis:

• •What magnitude of negative impact is acceptable?
• •What thresholds trigger concerns?
• •Can we monitor and revert if crossed?

Implementation:

• •Define acceptable thresholds
• •Set up automated alerts
• •Establish rollback criteria

Output:

## Mitigation Strategies

### Evaluated Strategies:

**Strategy 1: [Name]**
- Approach: [Description]
- Preserves: [Positive metrics]
- Addresses: [Negative metrics]
- Feasibility: [Easy/Medium/Hard]
- Timeline: [How long to implement]
- Risks: [What could go wrong]

**Strategy 2: [Name]**
[Same structure]

**Strategy 3: [Name]**
[Same structure]

### Recommended Mitigation:
**[Strategy name]**
- Why: [Rationale]
- Implementation: [Specific steps]
- Success criteria: [How to know it worked]
- Fallback plan: [If mitigation fails]

Step 4: Funnel Location Analysis (10 minutes)

Use the funnel-metric-mapping skill

Identify where in the user journey the trade-off occurs:

Questions:

1. •

   Which funnel stage is affected?

   • •Reach, Activation, Engagement, Retention?
   • •Multiple stages?
2. •

   Is impact concentrated or distributed?

   • •Single point of friction?
   • •Systemic shift across journey?
3. •

   Do metrics at different stages conflict?

   • •Earlier stage improved, later stage harmed?
   • •Suggests optimization for wrong outcome
4. •

   What transitions are affected?

   • •Conversion rates between stages changed?
   • •Drop-off points shifted?

Output:

## Funnel Impact Analysis

**Affected stages:**
- [Stage name]: [Metric change description]
- [Stage name]: [Metric change description]

**Transition impacts:**
- [Stage A → Stage B]: [Conversion rate change]

**Concentration:**
- [Single point / Distributed across journey]

**Insights:**
- [What funnel analysis reveals about trade-off]

Step 5: Make Recommendation (10 minutes)

Synthesize all analyses into clear recommendation:

Decision options:

1. •

   Ship Fully

   • •When: Net positive across segments, strategic alignment high
   • •Action: Roll out to 100%
   • •Monitoring: Track counter-metrics
2. •

   Ship to Segments

   • •When: Positive for some users, negative for others
   • •Action: Roll out to net-positive segments
   • •Plan: Iterate for problematic segments
3. •

   Iterate First

   • •When: Promising but needs modification
   • •Action: Implement mitigation strategy
   • •Timeline: Test modified version before full ship
4. •

   Gradual Rollout

   • •When: Uncertain about long-term effects
   • •Action: Staged rollout with monitoring
   • •Triggers: Rollback criteria defined
5. •

   Rollback

   • •When: Net negative, no clear mitigation, high risk
   • •Action: Revert to previous state
   • •Learning: Document why for future

Recommendation document:

# Trade-off Decision: [Change Name]

## Executive Summary
[One paragraph: Metrics moved, strategic assessment, recommendation]

## Metric Changes
**Positive:**
- [Metric]: +X% ([why this matters])
- [Metric]: +X% ([why this matters])

**Negative:**
- [Metric]: -X% ([why this matters])
- [Metric]: -X% ([why this matters])

## Strategic Assessment (Step 1)
**Most important metric:** [Name]
**Rationale:** [Why strategically critical]
**Alignment:** [High/Medium/Low]

## Temporal Analysis (Step 2)
**Short-term:** [Net positive/negative]
**Long-term:** [Net positive/negative]
**Trade-off type:** [Classification]

## Mitigation Options (Step 3)
**Explored:**
1. [Strategy name]: [Feasibility]
2. [Strategy name]: [Feasibility]

**Recommended:** [Strategy name]

## Funnel Impact (Step 4)
**Affected stage:** [Stage name]
**Insight:** [What this reveals]

## Recommendation

**Decision:** [Ship Fully / Ship to Segments / Iterate First / Gradual / Rollback]

**Rationale:**
- [Point 1]
- [Point 2]
- [Point 3]

**Implementation:**
- Action: [Specific steps]
- Owner: [Name/team]
- Timeline: [When]
- Success criteria: [How to measure]

**Risks:**
- Risk 1: [Description + mitigation]
- Risk 2: [Description + mitigation]

**Monitoring Plan:**
- Metrics to track: [List]
- Review frequency: [Daily/Weekly]
- Rollback triggers: [Conditions that require revert]
- Responsible: [Name/team]

## Appendices
- A/B test results: [Detailed data]
- Segment analysis: [Breakdown]
- Stakeholder input: [Summary]

Decision Matrix

Use this matrix for quick assessment:

Common Mistakes

Success Criteria

Tradeoff decision succeeds when:

• •Strategic alignment assessed (which metric matters more)
• •Temporal trade-offs evaluated (short vs. long-term)
• •3-5 mitigation strategies explored
• •Funnel impact analyzed
• •Clear recommendation made with rationale
• •Implementation plan defined
• •Risks identified with mitigation plans
• •Monitoring plan established
• •Stakeholders aligned on decision

Real-World Example: Snapchat Redesign

Context

• •Redesign launched: Separate social from media
• •Time on site: ↑ 15%
• •Messages sent: ↓ 12%
• •Stories shared: ↓ 8%

Step 1: Strategic Alignment (15 min)

Questions:

• •Which metric aligns with Snapchat's strategic positioning?
• •Camera/social app or content consumption app?

Analysis:

• •

  Content app positioning (time-on-site):

  • •Market saturated (TikTok, YouTube, Instagram)
  • •Not core competency
  • •Short-term thinking
• •

  Social/camera app positioning (messages/stories):

  • •Unique value proposition
  • •Core competency
  • •Sustainable competitive advantage

Conclusion: Messages and stories strategically more important than time-on-site

Step 2: Temporal Analysis (15 min)

Short-term (this quarter):

• •Revenue: Positive (more ads shown with higher time-on-site)
• •Growth: Neutral
• •User satisfaction: Mixed (some like creators, some miss social)

Long-term (12 months):

• •Retention: Negative risk (broken social flywheel)
  • •Fewer stories → Fewer notifications → Less app opening
  • •Fewer messages → Less reciprocal engagement
• •Flywheel: Weakened critical loop
• •Competitive position: Confused (competing with TikTok in saturated market)

Classification: Short-term gain (time/ads), long-term loss (retention/flywheel)

Assessment: Net negative

Step 3: Mitigation Strategies (25 min)

Strategy 1: Immediate Rollback

• •Pros: Restores social flywheel quickly
• •Cons: Loses time-on-site gains, admits failure
• •Feasibility: Easy

Strategy 2: Targeted Modifications

• •Approach: Keep creator content but re-integrate friend stories
• •Implementation:
  • •Show friend stories in creator tab
  • •Add "message friend" prompts after content viewing
  • •Balance algorithm between creator and friend content
• •Preserves: Time-on-site gains
• •Addresses: Social engagement losses
• •Feasibility: Medium complexity, 4-6 weeks

Strategy 3: Segmented Experience

• •Heavy social users: Friend-focused interface
• •Content consumers: Creator-focused interface
• •Feasibility: Complex, 8-12 weeks

Recommended: Strategy 2 (Targeted Modifications)

• •Best balance of preserving gains and fixing losses
• •Reasonable timeline
• •Testable before full rollout

Step 4: Funnel Analysis (10 min)

Affected stages:

• •Engagement (Depth): Messages and stories down (core actions)
• •Engagement (Breadth): Time on site up (passive consumption)

Insight:

• •Optimizing breadth at expense of depth
• •Depth drives retention flywheel, breadth doesn't
• •Classic mistake: Vanity metric over strategic metric

Step 5: Recommendation (10 min)

Decision: Iterate First (Don't immediately rollback, don't keep as-is)

Rationale:

1. •Clear strategic importance of social metrics
2. •Long-term retention risk outweighs short-term revenue gain
3. •Feasible mitigation strategy exists
4. •Can preserve some gains while fixing critical issue

Implementation:

• •Action: Implement targeted modifications (Strategy 2)
• •Owner: Product team + Design
• •Timeline: 4-6 weeks for modified version
• •Success criteria:
  • •Messages sent: Recover to within 5% of baseline
  • •Stories shared: Recover to within 5% of baseline
  • •Time on site: Maintain at least 50% of gains

Risks:

• •Risk: Modifications don't fix social engagement
  • •Mitigation: Full rollback if metrics don't improve in 30 days
• •Risk: Lose all time-on-site gains
  • •Mitigation: Acceptable trade-off for strategic alignment

Monitoring:

• •Daily: Messages, stories, time-on-site
• •Weekly: Full funnel review
• •Rollback trigger: If messages/stories drop further 5%

Timeline: 90 minutes for full analysis

Related Skills

This workflow orchestrates these skills:

• •north-star-alignment (Step 1)
• •tradeoff-evaluation (Steps 2-3)
• •funnel-metric-mapping (Step 4)

Related Workflows

• •metric-diagnosis: Use first if root cause of metric changes unclear
• •metrics-definition: Defines counter-metrics to detect trade-offs early
• •dashboard-design: Monitors trade-offs through balanced metric sets

Time Estimate

Total: 70-90 minutes

• •Step 1 (Strategic alignment): 15 min
• •Step 2 (Temporal analysis): 15 min
• •Step 3 (Mitigation): 20-30 min
• •Step 4 (Funnel analysis): 10 min
• •Step 5 (Recommendation): 10 min