Test Run Analysis

This skill helps you review Freeplay test run results, view evaluation metrics, and identify patterns across test cases.

When to use this skill

• •"Summarize the results of test run X"
• •"Summarize the major differences between two test runs in the same comparison"
• •"What were the metrics for my last test run?"
• •"Analyze test run [ID]"
• •"Which test cases failed in run X?"
• •"What are the evaluation scores from this test?"
• •User mentions a specific test run ID and wants insights
• •Compare two test runs and summarize differences between latency, cost, and evaluations
• •Select a winner from two test runs based on available data

Available API Endpoints

Get Test Run Summary

Endpoint: GET /api/v2/projects/{project_id}/test-runs/id/{test_run_id}

Returns:

{
  "id": "test-run-uuid",
  "name": "Test run name",
  "description": "Description",
  "created_at": 1234567890,
  "prompt_name": "Prompt template name",
  "prompt_version": "version-id",
  "model_name": "gpt-4",
  "sessions_count": 10,
  "summary_statistics": {
    "human_evaluation": {},
    "auto_evaluation": {
      "criteria_name": {
        "pass": 8,
        "fail": 2,
        "average_score": 0.85
      }
    },
    "client_evaluation": {}
  }
}

List All Test Runs

Endpoint: GET /api/v2/projects/{project_id}/test-runs

Returns a list of all test runs with their IDs and associated test case IDs.

How to Analyze a Test Run

When a user provides a test run ID, follow these steps:

1. Fetch Test Run Summary

Use curl or Python to call the API:

curl -H "Authorization: Bearer $FREEPLAY_API_KEY" \
     "$FREEPLAY_BASE_URL/api/v2/projects/{project_id}/test-runs/id/{test_run_id}"

2. Extract Key Insights

From the response, identify:

• •Overall performance: sessions_count, average scores
• •Evaluation metrics: Parse the summary_statistics object
  • •auto_evaluation: LLM-as-judge metrics
  • •human_evaluation: Human review scores
  • •client_evaluation: Custom evaluation criteria
• •Prompt details: prompt_name, prompt_version, model_name
• •Timestamp: created_at (Unix timestamp)

3. Analyze Metrics

For each evaluation criterion in summary_statistics:

• •Count pass/fail cases
• •Calculate pass rate percentage
• •Identify average scores
• •Flag criteria with low performance

Environment Variables Required

The following environment variables must be set:

• •FREEPLAY_API_KEY: Your Freeplay API key
• •FREEPLAY_BASE_URL: Freeplay API base URL (default: https://app.freeplay.ai)

These are typically configured in the plugin's .mcp.json file or user's environment.

Project ID can come from:

• •User specification
• •MCP list_projects() tool to discover available projects

Response Format

When presenting test run results to the user, include:

1. •

   Summary Header

   • •Test run name
   • •Prompt tested & version
   • •Model used
   • •Number of test cases
   • •Date/time
2. •

   Evaluation Metrics

   • •For each criterion:
     • •Criterion name
     • •Pass/fail count
     • •Pass rate percentage
     • •Average score (if applicable)
3. •

   Key Insights

   • •Overall pass rate
   • •Best performing criteria
   • •Areas needing improvement
   • •Any notable patterns or failures
4. •

   Recommendations

   • •Suggest next steps based on results
   • •Highlight specific test cases to review if pass rate is low
   • •When no evaluations are configured: The test run completed but has no quantitative metrics to analyze. Provide specific, actionable guidance:
     • •Explain that without evaluations, test runs only capture raw outputs — evaluations are what score those outputs automatically on each run
     • •Direct the user to set up evaluations: open the prompt in the Freeplay UI → go to the Evaluate tab → create evaluation criteria (e.g., model-graded criteria that check for accuracy, relevance, tone, or format compliance)
     • •Link to the documentation: Evaluations Overview
     • •Suggest running the health-check skill (/health-check) to get a full assessment of their project setup, including evaluation coverage
     • •For immediate value, recommend reviewing a sample of test case outputs directly in the Freeplay UI to manually spot-check quality before investing in automated evaluations

Example Analysis Output

# Test Run: "New test"
- Prompt: Chat Transcript to Issue
- Model: gpt-4-turbo
- Test Cases: 15
- Date: Jan 30, 2025

## Evaluation Results

### Accuracy (auto_evaluation)
- Pass: 13/15 (86.7%)
- Average Score: 0.87

### Relevance (auto_evaluation)
- Pass: 14/15 (93.3%)
- Average Score: 0.92

### Format Compliance (auto_evaluation)
- Pass: 15/15 (100%)
- Average Score: 1.0

## Overall Performance
✓ Strong performance across all criteria
✓ 93.3% overall pass rate
⚠ Review 2 failing test cases for Accuracy criterion

## Recommendations
1. Investigate the 2 accuracy failures to identify patterns
2. Consider adding more test cases for edge cases
3. Prompt is ready for staging deployment

Example Analysis Output (Multiple Test Runs)

# Test Run Comparison

## Test Run 1: "Bug Report Summarization"
- Prompt: Bug-to-Summary Converter
- Model: gpt-4-turbo
- Test Cases: 10
- Date: Jan 15, 2025

### Evaluation Results

#### Accuracy (auto_evaluation)
- Pass: 8/10 (80.0%)
- Average Score: 0.82

#### Clarity (manual_evaluation)
- Pass: 7/10 (70.0%)
- Average Score: 0.73

#### Format Compliance (auto_evaluation)
- Pass: 10/10 (100%)
- Average Score: 1.0

## Overall Performance
✓ Good accuracy and format compliance
⚠ Clarity underperforms at 70%
✓ 83.3% average pass rate

## Recommendations
1. Refine prompt instructions to improve clarity in summaries
2. Review the 3 test cases with clarity failures
3. Maintain strong format adherence

---

## Test Run 2: "Support Ticket Categorization 234dv vs a24bd"
- Prompt: Ticket Classifier
- Versions compared: 
    - Version: 234dv
      - Model: Opus 4.5
      - Created by: Prompt optimizer
      - Created: 10/11/25 11:12:32
    - Version: a24bd
      - Model: Sonnet 4.5
      - Created by: Rob Rhyne
      - Created: 10/10/25 11:12:32
- Test Cases: 12
- Date: Feb 2, 2025

### Evaluation Results

#### Accuracy (auto_evaluation)
- Pass: 9/12 (75.0%)
- Average Score: 0.79

#### Category Relevance (auto_evaluation)
- Pass: 8/12 (66.7%) ⚠
- Average Score: 0.68

#### Format Compliance (auto_evaluation)
- Pass: 11/12 (91.7%)
- Average Score: 0.95

## Overall Performance
⚠ Category relevance below 70% – needs attention
✓ Good format compliance, moderate accuracy
✗ 77.8% average pass rate

## Recommendations
1. Investigate category relevance failures—review misclassified tickets
2. Consider adding category examples to the prompt
3. Re-evaluate if accuracy improves after prompt update

Tips

• •Always format Unix timestamps into readable dates
• •Calculate percentages and round to 1 decimal place
• •Use visual indicators (✓ ⚠ ✗) for quick scanning
• •Highlight concerning metrics (< 70% pass rate)
• •Keep the analysis concise but actionable