Troubleshooter

Overview

This skill provides systematic troubleshooting methodology for SF-Bench issues. It covers evaluation failures, patch application errors, scratch org problems, and debugging strategies.

When This Skill Applies

• •Evaluation failures or unexpected errors
• •Patch application issues
• •Scratch org creation or deployment problems
• •Debugging test failures
• •Investigating root causes

Troubleshooting Methodology

Step 1: Categorize the Error

Step 2: Gather Evidence

1. •

   Check Logs

   bash

   # View recent logs
   ls -la logs/
   
   # Search for errors
   grep -r "ERROR" logs/
   
   # View specific task log
   cat logs/<run_id>/<model>/<task_id>.log
   

2. •

   Check Git Status

   bash

   # Verify no manual tweaks
   git status
   
   # View applied patch
   git diff
   

3. •

   Check Scratch Org

   bash

   # List orgs
   sf org list --all
   
   # View org details
   sf org display -o <alias>
   
   # Check deployment status
   sf project deploy report -o <alias>
   

Step 3: Identify Root Cause

Use the error message to narrow down:

Error Message → Category → Root Cause → Fix

Step 4: Apply Fix and Verify

1. •Apply the fix
2. •Run relevant tests
3. •Verify the issue is resolved
4. •Document if pattern-based

Common Error Patterns

Patch Application Failures

Error: "patch does not apply"

Cause: Patch context doesn't match file content

Diagnosis:

# Check file content
cat <file-path>

# Check patch preview
head -50 solution.patch

Solutions:

1. •Verify correct repository version
2. •Use fuzzy matching: git apply --3way
3. •Regenerate solution with updated context

Error: "corrupt patch at line X"

Cause: Malformed patch format

Diagnosis:

# Validate patch syntax
git apply --check solution.patch

Solutions:

1. •Check for markdown fences in patch
2. •Remove standalone +/- lines
3. •Verify patch header format

Scratch Org Failures

Error: "SCRATCH_ORG_LIMIT_EXCEEDED"

Cause: Daily scratch org creation limit reached

Diagnosis:

# Check org limits
sf limits api display -o <devhub>

Solutions:

1. •Delete unused scratch orgs
2. •Wait for daily limit reset
3. •Use different DevHub

Error: "InvalidOrgForCreate"

Cause: DevHub not authenticated or expired

Diagnosis:

# Check DevHub status
sf org list --all | grep DevHub

Solutions:

1. •Re-authenticate: sf org login web -d
2. •Verify DevHub features enabled
3. •Check org expiration

Deployment Failures

Error: "CANNOT_ENABLE_DEVELOPER_MODE"

Cause: Scratch org config issue

Diagnosis:

# Check scratch def
cat config/project-scratch-def.json

Solutions:

1. •Verify features in scratch def
2. •Check for conflicting settings
3. •Use minimal scratch def

Error: "Test coverage is XX%, at least 75% required"

Cause: Insufficient test coverage

Diagnosis:

# Run tests with coverage
sf apex run test -o <alias> -c -r human

Solutions:

1. •Add more test methods
2. •Cover edge cases
3. •Check for uncovered branches

API Failures

Error: "rate_limit_exceeded"

Cause: AI provider rate limit

Diagnosis:

• •Check request frequency
• •Review provider dashboard

Solutions:

1. •Implement rate limiting
2. •Use exponential backoff
3. •Switch to different provider

Error: "invalid_api_key"

Cause: Missing or incorrect API key

Diagnosis:

# Check environment
echo $ROUTELLM_API_KEY | head -c 10

Solutions:

1. •Verify key in .env
2. •Source environment: source .env
3. •Regenerate API key

Diagnostic Commands

Preflight Diagnostics

# Run preflight checks
python -c "
from sfbench.utils.preflight import PreflightValidator
v = PreflightValidator()
r = v.run_all_checks('model-name', 'provider')
print('PASSED' if r.passed else 'FAILED')
for c in r.checks:
    print(f'{c.name}: {c.status}')
"

Scratch Org Diagnostics

# Create debug org
sf org create scratch -f config/project-scratch-def.json -a debug-org -d 1

# Deploy and check
sf project deploy start -o debug-org

# Run specific test
sf apex run test -o debug-org -n TestClassName -r human -c

# Cleanup
sf org delete scratch -o debug-org -p

Patch Diagnostics

# Validate patch
git apply --check solution.patch

# Apply with verbose output
git apply -v solution.patch

# Apply with fallback strategies
git apply --3way solution.patch
git apply --reject solution.patch

Escalation Path

If issue persists after troubleshooting:

1. •

   Document the Issue

   • •Error message
   • •Steps to reproduce
   • •Environment details
   • •Attempted solutions
2. •

   Update activeContext.md

   • •Add to "Blockers" section
   • •Note impact on current work
3. •

   Create Issue

   • •Use bug report template
   • •Include all diagnostic information
   • •Tag appropriately

Prevention Strategies

For Patch Issues

• •Validate patches before applying
• •Use multi-strategy fallback
• •Clean patches before application

For Scratch Org Issues

• •Monitor org limits
• •Always cleanup in finally blocks
• •Use retry with exponential backoff

For API Issues

• •Implement rate limiting
• •Cache responses where appropriate
• •Use circuit breaker pattern for repeated failures