Superforecasting

Category: Decision-Making & Strategic Thinking Source: Philip Tetlock & Dan Gardner - "Superforecasting: The Art and Science of Prediction" (2015) Practitioner Score: 46/50 (Tier 1 Canonical)

Overview

Superforecasting is a systematic methodology for making accurate probabilistic predictions, developed through Philip Tetlock's Good Judgment Project. The research identified "superforecasters" - individuals who consistently outperform experts, pundits, and even intelligence analysts by 30% or more. The framework codifies their techniques into 10 actionable commandments plus deliberate practice protocols.

Core Insight: Prediction accuracy is a learnable skill. By combining rigorous process (break problems down, update beliefs incrementally, balance outside/inside views) with calibrated probabilistic thinking and error analysis, anyone can dramatically improve forecasting ability.

Evidence: Good Judgment Project participants trained in these techniques beat CIA analysts with classified information access.

When to Use

•Strategic decisions - Market entry, product launches, competitive moves
•Resource allocation - Investment decisions, hiring plans, capacity planning
•Risk assessment - Project timelines, crisis likelihood, threat analysis
•Competitive intelligence - Predicting competitor actions, market shifts
•Long-range planning - Technology adoption curves, regulatory changes

Anti-patterns:

•Decisions already made (confirmation bias trap)
•Binary yes/no thinking without probability ranges
•One-shot predictions without learning feedback
•High-emotion situations without cooling-off period

How to Execute

Step 1: Triage - Choose Forecast-Worthy Questions

Action: Focus effort on questions where hard work pays off

•Skip "clocklike" questions: Simple rules/trends suffice (e.g., "Will sun rise tomorrow?")
•Skip "cloud-like" questions: Too random even for models (e.g., "Will World War III start?")
•Target Goldilocks zone: Difficult but tractable with analysis
•Output: Prioritized list of forecastable questions

Step 2: Fermi-Style Decomposition

Action: Break intractable problems into tractable sub-problems

•Identify knowable parts: What can be researched or estimated?
•Expose unknowables: Flush ignorance into the open
•Example: "Will product X succeed?" → market size × conversion rate × pricing × competition
•Output: Hierarchical breakdown with researchable components

Step 3: Balance Outside View (Base Rates) and Inside View

Action: Start with reference class, adjust for specifics

•Outside view first: How often do things of this sort happen in situations of this sort?
•Find comparison class: Similar products, markets, technologies
•Inside view adjustment: What makes this case unique?
•Output: Base rate probability + reasoned adjustments

Step 4: Incremental Belief Updating

Action: Update forecasts as new evidence arrives - not too much, not too little

•Avoid under-reacting: Ignoring genuinely new information
•Avoid over-reacting: Jumping to conclusions from noisy signals
•Bayesian mindset: P(H|E) = P(E|H) × P(H) / P(E)
•Output: Revised probability with explicit reasoning

Step 5: Consider Clashing Causal Forces

Action: Map arguments for AND against your thesis

•Steel-man opposition: Understand counterarguments deeply
•Force interaction: How do conflicting factors balance?
•Example: "AI adoption" → Cost savings (pro) vs. Implementation complexity (con)
•Output: Two-column list of forces with relative weights

Step 6: Granular Probability Estimates

Action: Translate vague hunches into numeric probabilities

•Avoid vague language: "Likely" means what exactly?
•Use fine gradations: 55% vs. 60% forces precision
•Calibration practice: Track how often your 70% predictions come true
•Output: Numeric probability (e.g., 68%) with confidence range

Step 7: Balance Under/Overconfidence

Action: Manage trade-off between decisiveness and humility

•Calibration: Are your 80% predictions correct 80% of the time?
•Resolution: Can you distinguish 60% from 80% events?
•Avoid extremes: "Definitely" (99%+) and "No way" (1%-) rarely justified
•Output: Calibrated probability that neither overstates nor understates certainty

Step 8: Learn from Errors Without Hindsight Bias

Action: Analyze mistakes while resisting "I knew it all along"

•Pre-mortem: Before outcome, write why forecast might fail
•Post-mortem: After outcome, compare to pre-mortem (not current knowledge)
•Brier score tracking: Measure accuracy over time
•Output: Error log with root cause analysis

Step 9: Leverage Team Wisdom

Action: Master collaborative forecasting dynamics

•Perspective-taking: Reproduce others' arguments to their satisfaction
•Precision questioning: Help clarify without judgment
•Constructive confrontation: Disagree without being disagreeable
•Output: Team forecast incorporating diverse viewpoints

Step 10: Master the Error-Balancing Bicycle

Action: Treat commandments as guidelines requiring constant judgment

•No rigid rules: Every situation is unique
•Deliberate practice: Forecasting is skill built through repetition
•Feedback loops: Clear, unambiguous results inform learning
•Output: Continuous improvement trajectory

Real-World Examples

Good Judgment Project (2011-2015):

•Superforecasters beat intelligence analysts by 30%
•Ordinary people trained in these methods outperformed experts
•Result: Validated that forecasting is a learnable skill

Prediction Markets (Metaculus, Good Judgment Open):

•Calibrated forecasters consistently identify probability ranges
•Aggregated predictions outperform individual experts
•Result: Operational use in policy, business, research

Tech Industry Product Forecasting:

•Decompose adoption rates into addressable market × conversion × retention
•Update predictions as beta data arrives
•Result: Better resource allocation, realistic roadmaps

Integration Points

Complements:

•Brier Score: Measures superforecasting accuracy quantitatively
•Fermi Estimation: Powers Step 2 decomposition
•Bayes' Theorem: Mathematical foundation for belief updating
•Calibration: Essential skill for Step 7 confidence management
•Base Rate Analysis: Core of Step 3 outside view

Contrasts with:

•Expert Intuition: Systematic process vs. gut feel
•Punditry: Probabilistic humility vs. confident pronouncements
•Binary Thinking: 65% vs. "yes/no"

Common Pitfalls

Pitfall 1: Anchoring on Initial Estimate

•Warning sign: Forecast barely moves despite major news
•Fix: Explicit belief updating protocol after each information update

Pitfall 2: Ignoring Base Rates

•Warning sign: "This time is different" without evidence
•Fix: Always start with outside view reference class

Pitfall 3: Overconfidence in Extremes

•Warning sign: Many forecasts at 5% or 95%
•Fix: Force justification for extreme probabilities, track calibration

Pitfall 4: Confirmation Bias in Research

•Warning sign: Only seeking evidence supporting initial view
•Fix: Actively search for disconfirming evidence (Step 5)

Pitfall 5: No Feedback Loop

•Warning sign: Making predictions but never tracking outcomes
•Fix: Maintain prediction log with dates, probabilities, and resolutions

Validation Checklist

• Question is in Goldilocks zone (neither trivial nor impossible)
• Problem decomposed into researchable sub-components
• Base rate identified from reference class
• Both supporting and opposing forces mapped
• Probability is numeric and granular (not vague language)
• Calibration tracked over time (70% predictions = 70% accuracy)
• Forecast updated as new information arrives
• Pre-mortem written before outcome known
• Team input incorporated through structured dialogue

Key Metrics

Brier Score: Primary accuracy measure (0 = perfect, 2 = worst)

•Formula: (1/N) Σ(forecast - outcome)²
•Target: < 0.20 for well-calibrated forecaster

Calibration: Do your X% predictions happen X% of the time?

•Plot predicted probability vs. observed frequency
•Perfect calibration = diagonal line

Resolution: Can you distinguish different probability levels?

•Difference in outcomes between 60% and 80% forecasts
•Higher resolution = better discrimination