Evaluation Skill
Evaluate LLM outputs systematically with rubrics, handle non-determinism, and implement LLM-as-judge patterns.
Core Insight: The 95% Variance Finding
Research shows 95% of output variance comes from just two sources:
- •80% from prompt tokens (wording, structure, examples)
- •15% from random seed/sampling
Temperature, model version, and other factors account for only 5%.
Implication: Focus evaluation on prompt quality, not model tweaking.
What's Included
Examples (examples/)
- •Prompt comparison - A/B testing prompts with rubrics
- •Model evaluation - Comparing outputs across models
- •Regression testing - Detecting output degradation
Reference Guides (reference/)
- •Rubric design - Multi-dimensional evaluation criteria
- •LLM-as-judge - Using LLMs to evaluate LLM outputs
- •Statistical methods - Handling non-determinism
Templates (templates/)
- •Rubric templates - Ready-to-use evaluation criteria
- •Judge prompts - LLM-as-judge prompt templates
- •Test case format - Structured test case templates
Checklists (checklists/)
- •Evaluation setup - Before running evaluations
- •Rubric validation - Ensuring rubric quality
Key Concepts
1. Multi-Dimensional Rubrics
Don't use single scores. Break down evaluation into dimensions:
| Dimension | Weight | Criteria |
|---|---|---|
| Accuracy | 30% | Factually correct, no hallucinations |
| Completeness | 25% | Addresses all requirements |
| Clarity | 20% | Well-organized, easy to understand |
| Conciseness | 15% | No unnecessary content |
| Format | 10% | Follows specified structure |
2. Handling Non-Determinism
LLMs are non-deterministic. Handle with:
code
Strategy 1: Multiple Runs - Run same prompt 3-5 times - Report mean and variance - Flag high-variance cases Strategy 2: Seed Control - Set temperature=0 for reproducibility - Document seed for debugging - Accept some variation is normal Strategy 3: Statistical Significance - Use paired comparisons - Require 70%+ win rate for "better" - Report confidence intervals
3. LLM-as-Judge Pattern
Use a judge LLM to evaluate outputs:
code
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Prompt │────▶│ Test LLM │────▶│ Output │
└─────────────┘ └─────────────┘ └─────────────┘
│
▼
┌─────────────┐ ┌─────────────┐
│ Rubric │────▶│ Judge LLM │
└─────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ Score │
└─────────────┘
Best Practice: Use stronger model as judge (Opus judges Sonnet).
4. Test Case Design
Structure test cases with:
typescript
interface TestCase {
id: string
input: string // User message or context
expectedBehavior: string // What output should do
rubric: RubricItem[] // Evaluation criteria
groundTruth?: string // Optional gold standard
metadata: {
category: string
difficulty: 'easy' | 'medium' | 'hard'
createdAt: string
}
}
Evaluation Workflow
Step 1: Define Rubric
yaml
rubric:
dimensions:
- name: accuracy
weight: 0.3
criteria:
5: "Completely accurate, no errors"
4: "Minor errors, doesn't affect correctness"
3: "Some errors, partially correct"
2: "Significant errors, mostly incorrect"
1: "Completely incorrect or hallucinated"
Step 2: Create Test Cases
yaml
test_cases:
- id: "code-gen-001"
input: "Write a function to reverse a string"
expected_behavior: "Returns working reverse function"
ground_truth: |
function reverse(s: string): string {
return s.split('').reverse().join('')
}
Step 3: Run Evaluation
bash
# Run test suite python evaluate.py --suite code-generation --runs 3 # Output # ┌─────────────────────────────────────────────┐ # │ Test Suite: code-generation │ # │ Total: 50 | Pass: 47 | Fail: 3 │ # │ Accuracy: 94% (±2.1%) │ # │ Avg Score: 4.2/5.0 │ # └─────────────────────────────────────────────┘
Step 4: Analyze Results
Look for:
- •Low-scoring dimensions - Target for improvement
- •High-variance cases - Prompt needs clarification
- •Regression from baseline - Investigate changes
Grey Haven Integration
With TDD Workflow
code
1. Write test cases (expected behavior) 2. Run baseline evaluation 3. Modify prompt/implementation 4. Run evaluation again 5. Compare: new scores ≥ baseline?
With Pipeline Architecture
code
acquire → prepare → process → parse → render → EVALUATE
│
┌───────┴───────┐
│ Compare to │
│ ground truth │
│ or rubric │
└───────────────┘
With Prompt Engineering
code
Current prompt → Evaluate → Score: 3.2 Apply principles → Improve prompt New prompt → Evaluate → Score: 4.1 ✓
Use This Skill When
- •Testing new prompts before production
- •Comparing prompt variations (A/B testing)
- •Validating model outputs meet quality bar
- •Detecting regressions after changes
- •Building evaluation datasets
- •Implementing automated quality gates
Related Skills
- •
prompt-engineering- Improve prompts based on evaluation - •
testing-strategy- Overall testing approaches - •
llm-project-development- Pipeline with evaluation stage
Quick Start
bash
# Design your rubric cat templates/rubric-template.yaml # Create test cases cat templates/test-case-template.yaml # Learn LLM-as-judge cat reference/llm-as-judge-guide.md # Run evaluation checklist cat checklists/evaluation-setup-checklist.md
Skill Version: 1.0 Key Finding: 95% variance from prompts (80%) + sampling (15%) Last Updated: 2025-01-15