SKILL.md

--- frontmatter

name: ai-bias-auditor
description: Use when identifying fairness issues in AI systems. Use before deployment and periodically after. Produces bias assessment, fairness metrics, mitigation strategies, and audit documentation.

AI Bias Auditor

Overview

Identify and address fairness issues in AI systems. Conduct structured bias audits, measure disparate impact, and recommend mitigation strategies.

Core principle: Fairness is not optional. Proactive bias auditing protects users and the organization.

When to Use

•Before deploying AI system
•After significant model changes
•Periodic audits of production systems
•Responding to bias complaints
•Regulatory compliance requirements

Output Format

yaml

bias_audit:
  system: "[System name]"
  audit_date: "[YYYY-MM-DD]"
  auditor: "[Name/Team]"
  audit_type: "[Pre-deployment | Periodic | Complaint-driven]"
  
  scope:
    model_purpose: "[What the model does]"
    decision_type: "[Recommendation | Automation | Scoring]"
    affected_population: "[Who is impacted]"
    protected_attributes:
      - attribute: "[e.g., gender, race, age]"
        available_in_data: [true | false]
        proxy_risk: "[Potential proxies]"
  
  methodology:
    data_analyzed:
      training_data: [true | false]
      production_data: [true | false]
      sample_size: "[N]"
      time_period: "[Date range]"
    
    metrics_used:
      - metric: "[Metric name]"
        definition: "[How calculated]"
        threshold: "[Acceptable range]"
    
    techniques:
      - "[Disparate impact analysis]"
      - "[Intersectional analysis]"
  
  findings:
    summary: "[Overall assessment]"
    risk_level: "[High | Medium | Low | None detected]"
    
    by_attribute:
      - attribute: "[Protected attribute]"
        groups_compared: ["[Group A]", "[Group B]"]
        
        metrics:
          - metric: "[Metric name]"
            group_a: "[Value]"
            group_b: "[Value]"
            disparity: "[Ratio or difference]"
            threshold: "[Acceptable]"
            status: "[Pass | Fail | Review]"
        
        finding: "[Interpretation]"
        severity: "[High | Medium | Low | None]"
    
    intersectional:
      - groups: "[e.g., gender + age]"
        finding: "[What was found]"
        severity: "[Level]"
  
  root_cause_analysis:
    potential_sources:
      - source: "[Data | Model | Feature | Label]"
        description: "[How it introduces bias]"
        confidence: "[High | Medium | Low]"
  
  mitigation:
    recommendations:
      - recommendation: "[What to do]"
        priority: "[High | Medium | Low]"
        effort: "[Estimate]"
        expected_impact: "[How it helps]"
    
    if_deployed:
      - "[Monitoring requirement]"
      - "[Human oversight requirement]"
  
  compliance:
    regulations_considered: ["[Regulation]"]
    documentation_provided: ["[What's documented]"]
    
  verdict:
    deploy_recommendation: "[Approve | Approve with conditions | Do not deploy]"
    conditions: ["[If conditional approval]"]
    next_audit: "[When]"

Fairness Metrics

Group Fairness Metrics

Metric	Definition	Threshold
Demographic Parity	Equal positive prediction rates across groups	Ratio > 0.8
Equalized Odds	Equal TPR and FPR across groups	Difference < 0.1
Predictive Parity	Equal precision across groups	Ratio > 0.8
Calibration	Equal accuracy of probability estimates	Similar calibration curves

Disparate Impact Ratio

code

DI Ratio = (Favorable outcome rate for protected group) / 
           (Favorable outcome rate for majority group)

Interpretation:
- DI > 0.8: Generally acceptable (80% rule)
- DI 0.6-0.8: Needs review
- DI < 0.6: Likely problematic

Statistical Parity Difference

code

SPD = |P(Ŷ=1|A=0) - P(Ŷ=1|A=1)|

Interpretation:
- SPD < 0.05: Minimal disparity
- SPD 0.05-0.10: Moderate disparity
- SPD > 0.10: Significant disparity

Bias Sources

Source	Examples	Detection
Historical bias	Past discrimination encoded in labels	Compare to fair baseline
Representation bias	Some groups underrepresented	Check training data distribution
Measurement bias	Different measurement quality by group	Audit data collection
Aggregation bias	One model for heterogeneous groups	Test per-group performance
Proxy variables	Features correlated with protected attributes	Correlation analysis

Audit Process

Phase 1: Scope Definition

yaml

scoping:
  questions:
    - "What decisions does this system inform?"
    - "Who is affected by these decisions?"
    - "What are the relevant protected attributes?"
    - "What harm could result from biased decisions?"
    - "What legal/regulatory requirements apply?"

Phase 2: Data Analysis

yaml

data_analysis:
  steps:
    - "Document protected attribute distribution"
    - "Identify potential proxy variables"
    - "Check for missing data patterns by group"
    - "Analyze historical label quality by group"

Phase 3: Model Analysis

yaml

model_analysis:
  steps:
    - "Calculate fairness metrics by group"
    - "Perform intersectional analysis"
    - "Test on held-out diverse dataset"
    - "Analyze feature importance by group"

Phase 4: Reporting

yaml

reporting:
  elements:
    - "Executive summary with risk level"
    - "Detailed findings with evidence"
    - "Root cause hypotheses"
    - "Prioritized recommendations"
    - "Compliance documentation"

Mitigation Strategies

Pre-Processing

Strategy	When to Use
Resampling	Underrepresented groups
Reweighting	Imbalanced impact
Data augmentation	Limited diverse examples

In-Processing

Strategy	When to Use
Fairness constraints	Optimize with fairness objective
Adversarial debiasing	Remove protected info from embeddings
Regularization	Penalize group disparity

Post-Processing

Strategy	When to Use
Threshold adjustment	Equalize acceptance rates
Calibration	Align predictions by group
Human review	High-stakes edge cases

Documentation Requirements

For Regulatory Compliance

yaml

compliance_documentation:
  model_card:
    - "Intended use and limitations"
    - "Training data description"
    - "Fairness evaluation results"
    - "Known biases and mitigations"
  
  audit_trail:
    - "Audit methodology"
    - "Data sources examined"
    - "Metrics calculated"
    - "Findings and decisions"
  
  ongoing_monitoring:
    - "Metrics tracked post-deployment"
    - "Alert thresholds"
    - "Review frequency"

AI Bias Auditor

Overview

When to Use

Output Format

Fairness Metrics

Group Fairness Metrics

Disparate Impact Ratio

Statistical Parity Difference

Bias Sources

Audit Process

Phase 1: Scope Definition

Phase 2: Data Analysis

Phase 3: Model Analysis

Phase 4: Reporting

Mitigation Strategies

Pre-Processing

In-Processing

Post-Processing

Documentation Requirements

For Regulatory Compliance

Audit Checklist