LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

Do not use this skill when

•The task is unrelated to llm evaluation
•You need a different domain or tool outside this scope

Instructions

•Clarify goals, constraints, and required inputs.
•Apply relevant best practices and validate outcomes.
•Provide actionable steps and verification.
•If detailed examples are required, open resources/implementation-playbook.md.

Use this skill when

•Measuring LLM application performance systematically
•Comparing different models or prompts
•Detecting performance regressions before deployment
•Validating improvements from prompt changes
•Building confidence in production systems
•Establishing baselines and tracking progress over time
•Debugging unexpected model behavior

Core Evaluation Types

1. Automated Metrics

Fast, repeatable, scalable evaluation using computed scores.

Text Generation:

•BLEU: N-gram overlap (translation)
•ROUGE: Recall-oriented (summarization)
•METEOR: Semantic similarity
•BERTScore: Embedding-based similarity
•Perplexity: Language model confidence

Classification:

•Accuracy: Percentage correct
•Precision/Recall/F1: Class-specific performance
•Confusion Matrix: Error patterns
•AUC-ROC: Ranking quality

Retrieval (RAG):

•MRR: Mean Reciprocal Rank
•NDCG: Normalized Discounted Cumulative Gain
•Precision@K: Relevant in top K
•Recall@K: Coverage in top K

2. Human Evaluation

Manual assessment for quality aspects difficult to automate.

Dimensions:

•Accuracy: Factual correctness
•Coherence: Logical flow
•Relevance: Answers the question
•Fluency: Natural language quality
•Safety: No harmful content
•Helpfulness: Useful to the user

3. LLM-as-Judge

Use stronger LLMs to evaluate weaker model outputs.

Approaches:

•Pointwise: Score individual responses
•Pairwise: Compare two responses
•Reference-based: Compare to gold standard
•Reference-free: Judge without ground truth

Quick Start

python

from llm_eval import EvaluationSuite, Metric

# Define evaluation suite
suite = EvaluationSuite([
    Metric.accuracy(),
    Metric.bleu(),
    Metric.bertscore(),
    Metric.custom(name="groundedness", fn=check_groundedness)
])

# Prepare test cases
test_cases = [
    {
        "input": "What is the capital of France?",
        "expected": "Paris",
        "context": "France is a country in Europe. Paris is its capital."
    },
    # ... more test cases
]

# Run evaluation
results = suite.evaluate(
    model=your_model,
    test_cases=test_cases
)

print(f"Overall Accuracy: {results.metrics['accuracy']}")
print(f"BLEU Score: {results.metrics['bleu']}")

Automated Metrics Implementation

BLEU Score

python

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

def calculate_bleu(reference, hypothesis):
    """Calculate BLEU score between reference and hypothesis."""
    smoothie = SmoothingFunction().method4

    return sentence_bleu(
        [reference.split()],
        hypothesis.split(),
        smoothing_function=smoothie
    )

# Usage
bleu = calculate_bleu(
    reference="The cat sat on the mat",
    hypothesis="A cat is sitting on the mat"
)

ROUGE Score

python

from rouge_score import rouge_scorer

def calculate_rouge(reference, hypothesis):
    """Calculate ROUGE scores."""
    scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
    scores = scorer.score(reference, hypothesis)

    return {
        'rouge1': scores['rouge1'].fmeasure,
        'rouge2': scores['rouge2'].fmeasure,
        'rougeL': scores['rougeL'].fmeasure
    }

BERTScore

python

from bert_score import score

def calculate_bertscore(references, hypotheses):
    """Calculate BERTScore using pre-trained BERT."""
    P, R, F1 = score(
        hypotheses,
        references,
        lang='en',
        model_type='microsoft/deberta-xlarge-mnli'
    )

    return {
        'precision': P.mean().item(),
        'recall': R.mean().item(),
        'f1': F1.mean().item()
    }

Custom Metrics

python

def calculate_groundedness(response, context):
    """Check if response is grounded in provided context."""
    # Use NLI model to check entailment
    from transformers import pipeline

    nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")

    result = nli(f"{context} [SEP] {response}")[0]

    # Return confidence that response is entailed by context
    return result['score'] if result['label'] == 'ENTAILMENT' else 0.0

def calculate_toxicity(text):
    """Measure toxicity in generated text."""
    from detoxify import Detoxify

    results = Detoxify('original').predict(text)
    return max(results.values())  # Return highest toxicity score

def calculate_factuality(claim