AgentSkillsCN

calibration-vs-discrimination-trade-offs

当提升预测准确度需要在“预测与现实的契合度”与“不同结果间的区分能力”之间寻求平衡时

SKILL.md
--- frontmatter
name: calibration-vs-discrimination-trade-offs
description: When improving forecast accuracy requires balancing how well predictions match reality vs. how well they distinguish between outcomes
tags: [forecasting, prediction, decision-making, accuracy, superforecasting]

Calibration vs. Discrimination Trade-Offs

Overview

In forecasting and prediction, two dimensions of accuracy often trade off against each other: calibration (how well predicted probabilities match actual frequencies) and discrimination (how well predictions distinguish between different outcomes). A well-calibrated forecaster who says "70% chance" should be right 70% of the time. A discriminating forecaster can tell the difference between 30% and 70% events. Optimal performance requires balancing both, but improving one can hurt the other.

Core Concepts

Calibration

Do your probabilities match reality?

  • •If you say "80% confident" 100 times, are you right ~80 times?
  • •Measures whether predicted probabilities = actual frequencies
  • •Overconfidence: Saying 90% when only right 70%
  • •Underconfidence: Saying 60% when right 80%

Perfect calibration: For all predictions at X%, exactly X% come true.

Discrimination (Resolution)

Can you tell the difference between likely and unlikely events?

  • •Can you distinguish 90% probability events from 10% events?
  • •Measures variance in predictions
  • •High discrimination: Confidently predicts extremes (10%, 90%)
  • •Low discrimination: Everything clustered around 50% (uninformative)

Perfect discrimination: Always predict 100% for events that happen, 0% for those that don't.

The Trade-Off

Why you can't always have both:

  • •High discrimination with poor calibration: Overconfident extremes (90% predictions right only 60% of time)
  • •High calibration with poor discrimination: Safe but uninformative (everything is 50%)
  • •Sweet spot: Maximally discriminating while maintaining calibration

Execution Steps (Improving Both Dimensions)

1. Measure Current Performance

Calibration:

  • •Plot predicted probabilities vs. actual outcomes
  • •Calibration curve should hug the diagonal (45° line)
  • •Brier score captures both calibration and discrimination

Discrimination:

  • •Compare variance in predictions
  • •Area under ROC curve (AUC)
  • •How often did high-confidence predictions beat low-confidence?

Example: Track predictions over 100 forecasts, plot calibration curve.

2. Identify Bias Direction

  • •Overconfident: Calibration curve above diagonal (predicted 80%, actual 60%)
  • •Underconfident: Calibration curve below diagonal (predicted 60%, actual 80%)
  • •Overly hedged: All predictions near 50% (poor discrimination)

3. Adjust for Calibration

If overconfident:

  • •Regress toward 50% (moderate extreme predictions)
  • •Ask "What would make me wrong?"
  • •Track base rates more carefully

If underconfident:

  • •Push toward extremes when evidence is strong
  • •Trust pattern recognition
  • •Acknowledge uncertainty costs (hedging isn't free)

4. Improve Discrimination

  • •Seek better information: Distinguish strong vs. weak signals
  • •Identify drivers: What factors predict different outcomes?
  • •Decompose questions: Break complex forecasts into sub-components
  • •Track leading indicators: Early signals that differentiate

Example: Instead of "Will product succeed?" ask "Will it hit X downloads AND Y retention?"

5. Balance the Trade-Off

Extremizing: When aggregating forecasts, push crowd average toward extremes (improves discrimination while maintaining calibration)

Granular confidence: Use 1-99% scale, not just 25/50/75

Context-dependent: High-stakes decisions need calibration; exploratory decisions can tolerate discrimination focus

Anti-Patterns

False Precision: Claiming 73% when you mean "probably" (discrimination theater without calibration)

Perpetual Hedging: Always saying 50-60% to avoid being wrong (good calibration, useless discrimination)

Uncalibrated Extremes: Bold predictions (10%, 90%) without tracking accuracy (discrimination without calibration)

Ignoring Base Rates: Overweighting anecdotes vs. statistical priors (poor calibration)

Quality Indicators

High Signal (Good Balance):

  • •Brier score < 0.20 (combines calibration + discrimination)
  • •Calibration curve near diagonal across full probability range
  • •Predictions vary meaningfully (not clustered at 50%)
  • •Confidence correlates with accuracy
  • •Regular scoring and feedback

Low Signal:

  • •Never track actual outcomes vs. predictions
  • •All predictions in narrow range (40-60%)
  • •Wildly overconfident (90% predictions right 50% of time)
  • •No improvement over time despite feedback

Cross-Domain Applications

Superforecasting

Philip Tetlock's research: Best forecasters balance both dimensions

  • •Track predictions in prediction markets or tournaments
  • •Use granular probabilities (not just high/medium/low)
  • •Update incrementally as new information arrives

Machine Learning

Model evaluation trade-offs:

  • •Precision vs. recall (similar to calibration vs. discrimination)
  • •Confidence scores should match actual accuracy
  • •Platt scaling: Post-hoc calibration of model outputs

Medical Diagnosis

  • •Discrimination: Can test distinguish sick from healthy?
  • •Calibration: Does "70% risk" mean 70 out of 100 similar patients?
  • •Both matter: Wrong treatment OR unnecessary anxiety

Business Forecasting

  • •Revenue predictions need calibration (for budgeting)
  • •Opportunity prioritization needs discrimination (which bets to make?)

Related Frameworks

  • •Brier Score: Combines calibration and discrimination in single metric
  • •Superforecasting: Tetlock's research on prediction accuracy
  • •Bayesian Updating: Incremental belief revision improves calibration
  • •Base Rate Neglect: Ignoring priors hurts calibration
  • •Extremizing: Aggregation technique that improves discrimination

Scoring (35/50)

  • •Practitioner Weight (7/10): Core to Tetlock's forecasting research, used in prediction markets
  • •Clarity (7/10): Concepts clear but measuring them requires statistical knowledge
  • •Proven ROI (8/10): Superforecasters demonstrably outperform by balancing both
  • •Novelty (6/10): Statistical concepts applied to forecasting (moderately non-obvious)
  • •Applicability (7/10): Relevant to forecasting, ML, risk assessment, decision-making

Sources

  • •Philip Tetlock: Superforecasting (calibration-discrimination trade-offs in expert predictions)
  • •Philip Tetlock: Expert Political Judgment (foxes vs. hedgehogs, accuracy dimensions)
  • •Glenn Brier: Verification of forecasts (Brier score)
  • •Good Judgment Project: Practical forecasting tournament findings
  • •Nate Silver: The Signal and the Noise (calibration in prediction)