Results Audit — Authenticity & Statistical Validity

Automated Red Flag Scan

# Quick anomaly detection
python3 -c "
import json, sys

results_file = 'workspace/results/iteration_001/test_results.json'  # adjust iteration
try:
    with open(results_file) as f:
        r = json.load(f)

    flags = []

    # Check for perfect metrics
    for k, v in r.items():
        if isinstance(v, float) and v >= 1.0:
            flags.append(f'SUSPICIOUS: {k} = {v} (perfect score)')
        if isinstance(v, float) and v == 0.0:
            flags.append(f'SUSPICIOUS: {k} = {v} (zero)')

    # Check seed is recorded
    if 'seed' not in r:
        flags.append('MISSING: random seed not recorded')

    if flags:
        print('🚩 RED FLAGS:')
        for f in flags:
            print(f'  - {f}')
    else:
        print('✅ No obvious red flags')
except Exception as e:
    print(f'❌ Cannot read results: {e}')
"

Verification Checklist

1. Training Log Integrity

• • training_log.json exists and is complete
• • Loss curve is monotonically decreasing (overall trend)
• • No sudden jumps that suggest training restart without logging
• • Number of epochs matches config
• • Timestamps are sequential (not fabricated)

2. Results Plausibility

Read expected range from workspace/logs/strategy_matrix.json:

• • Result within backward-induction expected range?
• • If result > expected + 5%: investigate data leak or bug
• • If result < expected - 10%: strategy may be failing
• • Variance across seeds is realistic (typically 0.5-3% for classification)

3. Cross-Consistency

• • test_results.json numbers match final epoch in training_log.json
• • Numbers in comparison_table.tex match comparison_results.json
• • Baseline numbers match their cited source (paper table/figure number)

4. Statistical Significance

For each "our method vs baseline" comparison:

• • Paired t-test or Wilcoxon computed (p < 0.05)
• • Multiple seeds used (minimum 3, recommended 5)
• • Mean AND standard deviation reported
• • No cherry-picked seeds

5. Reproducibility

• • Random seeds listed
• • requirements.txt complete
• • Data preprocessing is deterministic
• • Hardware info recorded

IMMEDIATE FAIL Conditions

Any of these → status = CRITICAL, pipeline halts:

• •Results too good (>5% above ALL baselines simultaneously)
• •Zero variance across multiple runs
• •Missing training logs entirely
• •Metrics that are mathematically impossible (e.g., precision > 1.0)
• •Baseline numbers don't match their original papers

Output Format

Write to workspace/logs/results_audit.json:

{
  "timestamp": "...",
  "iteration": N,
  "status": "PASS | WARN | CRITICAL",
  "expected_range": [low, high],
  "actual_result": X,
  "plausibility": "PLAUSIBLE | SUSPICIOUS | OUTSIDE_RANGE",
  "statistical_validity": "PASS | FAIL",
  "reproducibility": "CONFIRMED | UNCONFIRMED",
  "red_flags": [],
  "score": X
}