Extract and Summarize Test Failures

Parse test logs to extract failure information and create summary.

When to Use

• •CI test run failed, need to understand failures
• •Large test suite with many failures, need summary
• •Categorizing failures by type (assertion, runtime, timeout)
• •Creating PR review comments about failures
• •Tracking which tests are failing over time

Quick Reference

# Extract failed test names
grep "FAILED" test_output.log

# Get failure details with context
grep -A 10 "FAILED\|Error\|AssertionError" test_output.log

# Count failures by type
grep "FAILED\|AssertionError\|ValueError\|TypeError" test_output.log | sort | uniq -c

# Extract test summary line
tail -20 test_output.log | grep -E "passed|failed|error"

# Get specific failure info
grep -B 5 "AssertionError" test_output.log | head -50

Failure Extraction Workflow

1. •Collect log: Get test output from CI or local run
2. •Find failures: Extract FAILED markers and error types
3. •Group by type: Categorize assertion vs runtime vs timeout
4. •Extract details: Get error messages and stack traces
5. •Count totals: Summary of how many failures
6. •Identify patterns: Common failure causes
7. •Summarize: Create concise failure report

Failure Categories

Assertion Failures:

• •AssertionError - Expected value check failed
• •assert_equal() - Values don't match
• •assert_true() - Condition not true
• •Fix: Check test logic and expected values

Type/Attribute Errors:

• •AttributeError - Missing method/attribute
• •TypeError - Wrong argument type
• •ValueError - Invalid value for operation
• •Fix: Check code syntax and types

Runtime Errors:

• •IndexError - Out of bounds access
• •KeyError - Dictionary key not found
• •ZeroDivisionError - Division by zero
• •Fix: Check boundary conditions

Timeout/Hanging:

• •Test takes too long
• •Infinite loop or deadlock
• •Resource exhaustion
• •Fix: Optimize or add timeout

Output Format

Report failures with:

1. •Total Failures - Count and percentage of total tests
2. •Failure Summary - By type (assertion, runtime, timeout, other)
3. •Failed Tests - List of test names that failed
4. •Top Issues - Most common failure patterns
5. •Error Messages - Representative error snippets
6. •Recommendations - Which tests to focus on first

Error Handling

References

• •See test-diff-analyzer for identifying flaky tests
• •See analyze-ci-failure-logs for CI-specific failures
• •See generate-fix-suggestions for automated fixes