Dataset: Ground truth test set (fixed)
When: During development
Metrics: Accuracy, F1, BLEU, RAG metrics
Environment: Development/staging

Dataset: Live traffic (dynamic)
When: In production
Metrics: User satisfaction, task success, engagement
Environment: Production

Pros:
- Fast (minutes to hours)
- Reproducible (same dataset)
- Safe (no user impact)
- Cheap (no production traffic)

Use for:
- Rapid development
- Comparing models
- Regression testing

Pros:
- Real performance (actual users)
- Actual impact (business metrics)
- Catches issues offline can't (latency, UX)

Use for:
- Final validation
- Measuring business impact
- Continuous monitoring

Offline: "Model A is 5% more accurate"
Online: "Model A increases user satisfaction by 10%"

Both needed to make informed decisions

1. Create evaluation dataset (ground truth)
2. Run model on dataset
3. Compute metrics
4. Compare to baseline
5. Iterate until metrics improve
6. Deploy to production

Control (A): Current model (50% of users)
Treatment (B): New model (50% of users)

Random assignment of users
Measure metrics for both
Statistical significance test
Ship winner

import random

def assign_variant(user_id):
    if hash(user_id) % 2 == 0:
        return "A"  # Control
    else:
        return "B"  # Treatment

# Serve model based on variant
variant = assign_variant(user_id)
if variant == "A":
    answer = model_a.predict(question)
else:
    answer = model_b.predict(question)

# Log result
log_result(user_id, variant, question, answer, user_feedback)

from scipy.stats import ttest_ind

satisfaction_a = [4, 5, 3, 4, 5, ...]  # User ratings for A
satisfaction_b = [5, 5, 4, 5, 4, ...]  # User ratings for B

t_stat, p_value = ttest_ind(satisfaction_a, satisfaction_b)

if p_value < 0.05:
    print("Statistically significant difference!")
    if mean(satisfaction_b) > mean(satisfaction_a):
        print("Ship variant B")
else:
    print("No significant difference")

Production: Serve model A (current)
Shadow: Run model B in background (don't serve)
Compare: Model B predictions vs model A predictions
No user impact: Users only see model A

# Serve current model
answer_a = model_a.predict(question)
serve_to_user(answer_a)

# Run new model in shadow mode (async)
async def shadow_predict():
    answer_b = model_b.predict(question)
    log_shadow_result(question, answer_a, answer_b)
    
    # Compare
    if answer_a != answer_b:
        log_difference(question, answer_a, answer_b)

asyncio.create_task(shadow_predict())

1. Deploy new model to 1% of traffic
2. Monitor closely (errors, latency, satisfaction)
3. If good, increase to 5%
4. Gradually increase to 100%
5. Rollback if issues

def get_model(user_id):
    # Canary: 5% of users get new model
    if hash(user_id) % 100 < 5:
        return model_b  # New model (canary)
    else:
        return model_a  # Current model

model = get_model(user_id)
answer = model.predict(question)

# Monitor metrics
monitor_metrics(model_version=model.version, user_id=user_id)

# If error rate > 5% for canary
if canary_error_rate > 0.05:
    rollback_to_previous_version()
    alert_team("Canary deployment failed")

Show results from both models, interleaved
Track which results users click
Infer which model is better

Model A results: [R1_A, R2_A, R3_A, R4_A]
Model B results: [R1_B, R2_B, R3_B, R4_B]

Interleaved: [R1_A, R1_B, R2_A, R2_B, R3_A, R3_B]

User clicks: R1_B, R2_A
→ Model B: 1 click, Model A: 1 click

User clicks result → Likely relevant
User doesn't click → Likely not relevant

User reformulates → Dissatisfied with answer
User doesn't reformulate → Satisfied

User abandons → Failed to help
User continues → Successful

Long session → Engaged
Short session → Not engaged (or very efficient)

feedback = {
    "question": "What is the capital of France?",
    "answer": "Paris",
    "feedback": "thumbs_up",  # or "thumbs_down"
    "timestamp": "2024-01-15T10:00:00Z"
}

# Aggregate
thumbs_up_rate = thumbs_up / (thumbs_up + thumbs_down)

rating = {
    "question": "What is the capital of France?",
    "answer": "Paris",
    "rating": 5,  # 1-5 scale
    "timestamp": "2024-01-15T10:00:00Z"
}

# Aggregate
avg_rating = sum(ratings) / len(ratings)

feedback = {
    "question": "What is the capital of France?",
    "answer": "Paris",
    "feedback_text": "Perfect answer, very helpful!",
    "timestamp": "2024-01-15T10:00:00Z"
}

# Analyze sentiment
sentiment = analyze_sentiment(feedback_text)

bug_report = {
    "question": "What is the capital of France?",
    "answer": "Lyon",  # Incorrect
    "issue": "Incorrect answer",
    "timestamp": "2024-01-15T10:00:00Z"
}

# Track and fix
add_to_ground_truth(question, correct_answer="Paris")

Hypothesis: Higher offline accuracy → Higher online satisfaction

Test:
- Model A: 85% offline accuracy → 75% thumbs up
- Model B: 90% offline accuracy → 80% thumbs up

Correlation: ✓ Offline predicts online

Process:
1. Improve offline metric (85% → 90% accuracy)
2. Deploy to production (A/B test)
3. Measure online metric (75% → 80% thumbs up)
4. If online improves → Offline metric is good proxy
5. If online doesn't improve → Offline metric is misleading

Workflow:
1. Offline: Test 10 model variants
2. Filter: Keep top 3 (based on offline metrics)
3. Online: A/B test top 3
4. Ship: Best performer online

Offline: Model A is better (90% vs 85% accuracy)
Online: Model B performs better (80% vs 75% thumbs up)

Test set: Simple questions
Production: Complex, ambiguous questions
→ Test set doesn't match reality

Offline: Accuracy (correct answer)
Online: Helpfulness (useful answer)
→ Correct ≠ helpful

Test set: Factual questions
Production: Conversational queries
→ Different use cases

Online metrics = actual user impact
→ Trust online

But investigate:
- Why did offline mislead?
- Update offline dataset/metrics
- Improve offline-online correlation

log_entry = {
    "timestamp": "2024-01-15T10:00:00Z",
    "question": "What is the capital of France?",
    "answer": "Paris",
    "model_version": "v1.2.0",
    "latency_ms": 250,
    "user_feedback": "thumbs_up",
    "user_id": "user123"
}

db.logs.insert(log_entry)

# Weekly: Evaluate on last week's data
recent_data = db.logs.find({
    "timestamp": {"$gte": one_week_ago}
}).limit(1000)

# Evaluate
results = evaluate_model(recent_data)

# Compare to baseline
if results["accuracy"] < baseline_accuracy - 0.05:
    alert("Model performance degraded!")

Monthly: A/B test new model variant
Measure: User satisfaction, task success
Ship: If statistically significant improvement

Dashboard:
- Offline metrics (accuracy, F1)
- Online metrics (thumbs up rate, task success)
- Latency (P50, P95, P99)
- Error rate
- Traffic volume

Alerts:
- Accuracy drops >5%
- Thumbs up rate drops >10%
- Latency P95 >1s
- Error rate >1%

def monitor_canary():
    while True:
        metrics = get_canary_metrics()
        
        if metrics["error_rate"] > 0.05:
            rollback()
            alert("High error rate, rolled back")
        
        if metrics["thumbs_down_rate"] > 0.3:
            rollback()
            alert("High dissatisfaction, rolled back")
        
        time.sleep(60)  # Check every minute

Process:
1. Canary to 1% (automated)
2. Monitor for 24 hours
3. Manual review (check logs, user feedback)
4. If good, increase to 10%
5. Repeat until 100%

Start small:
- 1% canary (low risk)
- Monitor closely
- Gradually increase
- Never test unproven model on 100% traffic

Feature flags:
- Easy to turn off new model
- Instant rollback
- No code deployment needed

Iterate on model
Evaluate on test set (1000 examples)
Improve until offline metrics good

Run in shadow mode (1 week)
Evaluate on live traffic (offline metrics)
Compare to current model

Deploy to 1% of users
Monitor online metrics (24 hours)
If good, proceed

1% → 5% → 10% → 25% → 50% → 100%
Monitor at each step
Rollback if issues

Track metrics in production
Detect regressions
A/B test improvements

Offline: NDCG@10 (ranking quality)
Online: Click-through rate (CTR)

Correlation: High NDCG → High CTR

Offline: Precision@k, recall@k
Online: Engagement (clicks, watch time)

Correlation: High precision → High engagement

Offline: Faithfulness, relevance
Online: Thumbs up rate, task success

Correlation: High faithfulness → High thumbs up