Data Readiness Assessor

Overview

Evaluate whether data is ready for AI/ML projects before committing resources. Assess quality, availability, labeling needs, and identify gaps that require remediation.

Core principle: Data is the foundation. A thorough assessment prevents "garbage in, garbage out" and project delays.

When to Use

•Starting new AI/ML project
•Evaluating feasibility of AI use case
•Diagnosing model performance issues
•Planning data infrastructure investments

Output Format

yaml

data_assessment:
  project: "[Project name]"
  assessment_date: "[YYYY-MM-DD]"
  assessor: "[Name]"
  
  overall_readiness:
    score: "[1-5]"
    verdict: "[Ready | Ready with caveats | Not ready]"
    summary: "[Brief assessment]"
  
  data_sources:
    - source: "[Data source name]"
      type: "[Structured | Unstructured | Semi-structured]"
      location: "[Where stored]"
      owner: "[Data owner]"
      access: "[How to access]"
      
      volume:
        records: "[Count]"
        size: "[GB/TB]"
        time_range: "[Date range covered]"
        sufficient: "[Yes | No | Borderline]"
      
      quality:
        completeness:
          score: "[1-5]"
          missing_rate: "[%]"
          critical_fields_missing: ["[Field]"]
        
        accuracy:
          score: "[1-5]"
          known_issues: ["[Issue]"]
          validation_method: "[How verified]"
        
        consistency:
          score: "[1-5]"
          duplicates: "[% or count]"
          format_issues: ["[Issue]"]
        
        timeliness:
          score: "[1-5]"
          freshness: "[How recent]"
          update_frequency: "[How often updated]"
      
      relevance:
        features_available: ["[Feature 1]", "[Feature 2]"]
        features_missing: ["[Needed but not present]"]
        target_variable: "[Available | Derivable | Missing]"
  
  labeling_assessment:
    required: [true | false]
    current_state:
      labeled_volume: "[Count or %]"
      label_quality: "[High | Medium | Low | Unknown]"
      labeling_consistency: "[Assessment]"
    
    gap:
      additional_labels_needed: "[Count]"
      estimated_effort: "[Hours/days]"
      labeling_approach: "[Manual | Semi-automated | Crowdsourced]"
  
  integration:
    accessibility:
      api_available: [true | false]
      export_options: ["[Format options]"]
      real_time_possible: [true | false]
    
    legal_compliance:
      pii_present: [true | false]
      consent_status: "[Covered | Needs review | Not covered]"
      retention_policies: "[Compliant | Needs review]"
      cross_border: "[Applicable | Not applicable]"
  
  gaps:
    critical:
      - gap: "[Gap description]"
        impact: "[How it affects project]"
        remediation: "[How to fix]"
        effort: "[Time/cost estimate]"
    
    important:
      - gap: "[Gap description]"
        impact: "[How it affects project]"
        remediation: "[How to fix]"
  
  recommendations:
    proceed_if:
      - "[Condition for proceeding]"
    
    actions_required:
      - action: "[Required action]"
        owner: "[Who]"
        timeline: "[When]"
        blocking: [true | false]

Quality Dimensions

The Five V's Assessment

Dimension	Questions	Scoring
Volume	Enough data to train? Enough for validation?	5=Abundant, 1=Insufficient
Variety	Covers all scenarios? Edge cases represented?	5=Comprehensive, 1=Narrow
Velocity	Can get fresh data? Update frequency sufficient?	5=Real-time, 1=Stale
Veracity	How accurate? How consistent? Trust level?	5=Highly trusted, 1=Unreliable
Value	Contains needed features? Labels available?	5=Complete, 1=Lacking

Sample Size Guidelines

Model Type	Minimum Samples	Recommended
Simple classification	100 per class	1,000+ per class
Complex classification	1,000 per class	10,000+ per class
Regression	100-1,000	10,000+
Deep learning	10,000+	100,000+
LLM fine-tuning	100-1,000 examples	10,000+

Data Quality Scorecard

yaml

quality_scorecard:
  dimension: "Completeness"
  scoring:
    5: "<1% missing values in critical fields"
    4: "1-5% missing values, no critical gaps"
    3: "5-15% missing values, some critical gaps"
    2: "15-30% missing values, significant gaps"
    1: ">30% missing or critical fields unavailable"
  
  dimension: "Accuracy"
  scoring:
    5: "Validated against ground truth, <1% error"
    4: "Spot-checked, <5% error rate"
    3: "Some validation, known issues documented"
    2: "Limited validation, suspected issues"
    1: "No validation, reliability unknown"
  
  dimension: "Consistency"
  scoring:
    5: "Standardized formats, no duplicates"
    4: "Minor format variations, <1% duplicates"
    3: "Multiple formats, 1-5% duplicates"
    2: "Significant format issues, 5-10% duplicates"
    1: "Major inconsistencies, >10% duplicates"

Labeling Assessment

Labeling Quality Checklist

yaml

labeling_quality:
  guidelines:
    - "Clear labeling instructions exist"
    - "Edge cases documented"
    - "Examples provided for each class"
  
  process:
    - "Multiple labelers for quality"
    - "Inter-annotator agreement measured"
    - "Disagreements have resolution process"
  
  coverage:
    - "All classes represented"
    - "Class distribution acceptable"
    - "Edge cases labeled"

Labeling Effort Estimation

Complexity	Time per Item	Items per Hour
Binary classification	5-10 sec	360-720
Multi-class (5-10 classes)	15-30 sec	120-240
Complex annotation	1-5 min	12-60
Expert annotation	5-30 min	2-12

Red Flags

Red Flag	Implication	Response
No access to raw data	Can't validate quality	Negotiate access or find alternative
Unknown data lineage	Reliability questionable	Trace source, validate sample
PII without consent	Legal/compliance risk	Legal review required
Single source only	No validation possible	Find corroborating source
Labels from same source as features	Leakage risk	Separate label source
Highly imbalanced classes	Model bias risk	Plan for oversampling/weighting

Readiness Levels

Level	Score	Meaning	Action
Ready	4-5	Proceed with project	Begin development
Ready with caveats	3	Proceed with mitigation	Address gaps in parallel
Not ready	1-2	Do not proceed yet	Remediate before starting

Assessment Checklist

• All data sources identified
• Access verified for each source
• Volume sufficiency assessed
• Quality dimensions scored
• Labeling needs determined
• Legal/compliance reviewed
• Gaps documented with remediation
• Readiness verdict provided