Test Standards

Validates test quality through automated scanning and LLM-guided heuristics to detect:

• •Mesa-optimization (tests that pass trivially)
• •No assertions (tests execute but verify nothing)
• •Tautological assertions (always true regardless of implementation)
• •Vacuous property checks (only existence, not values)
• •Mock-only validation (only verify mocks called, not behavior)
• •Happy-path bias (only success tests, no error handling)
• •Error-swallowing patterns (catch blocks without assertions)
• •Missing async test protection (expect.assertions(n))
• •Low assertion density (superficial testing)

When to Use

Execute test-standards during QA validation:

1. •After reviewing test file changes
2. •When tests were modified by Action Agent
3. •Before approving changes for merge
4. •When evaluating test meaningfulness
5. •During test creation/maintenance

Validation Workflow

1. Automated Scanning (Script)

Run script first for deterministic, fast detection:

Scan specific test files:

python scripts/test_quality_scanner.py tests/auth.test.ts tests/api.test.ts --format json

Scan test directory:

python scripts/test_quality_scanner.py ./tests --format json

Save report:

python scripts/test_quality_scanner.py ./tests --output test-quality-report.json

Exclude patterns:

python scripts/test_quality_scanner.py ./tests --exclude node_modules .snapshots --format json

2. Interpret Scan Results

Parse JSON output and evaluate findings:

Finding Structure:

{
  "category": "no_assertions|tautological_assertion|vacuous_check|mock_only_validation|...",
  "severity": "CRITICAL|HIGH|MEDIUM|LOW",
  "file": "tests/auth.test.ts",
  "test_name": "validates user login",
  "line": 42,
  "pattern": "regex pattern or description",
  "context": "relevant code snippet",
  "message": "human-readable description"
}

Report Structure:

{
  "files_scanned": 15,
  "total_tests": 128,
  "total_findings": 12,
  "findings_by_severity": {
    "CRITICAL": 2,
    "HIGH": 5,
    "MEDIUM": 4,
    "LOW": 1
  },
  "findings": [...],
  "test_stats": [
    {
      "file": "tests/auth.test.ts",
      "total_tests": 25,
      "tests_with_assertions": 23,
      "tests_without_assertions": 2,
      "async_tests_without_protection": 3,
      "error_tests": 5,
      "success_tests": 18,
      "avg_assertions_per_test": 3.2
    }
  ],
  "summary": {
    "no_assertions": 2,
    "tautological_assertions": 1,
    "vacuous_checks": 3,
    "mock_only_validation": 2,
    "missing_assertion_count": 3,
    "error_swallowing": 1,
    "low_assertion_density": 0,
    "happy_path_bias": 1
  }
}

Severity Guidelines:

• •CRITICAL: No assertions, tautological assertions - BLOCK merge immediately
• •HIGH: Vacuous checks, mock-only validation, error swallowing - Require fixes
• •MEDIUM: Happy-path bias, missing assertion count, low density - Review and justify
• •LOW: Minor quality issues - Optional fixes

3. LLM Heuristic Review (Context-Dependent)

After automated scanning, apply judgment for patterns the script cannot detect:

Test Assertion Weakening

Compare test changes in git diff to detect semantic weakening:

Indicators of Weakening:

• •Reduced assertion count without clear reason
• •Replaced specific assertions with generic checks
• •Removed edge case validations
• •Changed from behavior validation to mock validation only

Example - Weakened Test:

// Before (git diff -)
expect(response.data).toMatchObject({
  id: expect.any(String),
  status: 'active',
  count: expect.any(Number)
});

// After (git diff +)
expect(response.data).toBeDefined(); // ❌ Lost specificity

Action: Flag as HIGH severity, request Action Agent restore specific assertions.

Scope Alignment

Verify test changes align with issue scope:

Legitimate Changes:

• •Tests added for new feature functionality
• •Tests updated to match changed implementation
• •Error handling tests added for new edge cases

Suspicious Changes:

• •Unrelated tests modified without explanation
• •Tests removed without justification
• •Test patterns changed across multiple files beyond issue scope

Action: Request clarification or justification in scratch notes.

Test Coverage Gaps

Identify functions/features lacking tests:

Review Approach:

1. •Read implementation changes from git diff
2. •Identify new functions, branches, error paths
3. •Check if corresponding tests exist
4. •Flag missing coverage for critical paths

Action: Request additional tests for uncovered critical functionality.

4. Generate Validation Report

Combine automated findings with heuristic review:

## Test Standards Validation for [ISSUE-ID]

### Automated Scan Summary
- Test Files Scanned: X
- Total Tests: Y
- Total Findings: Z
- CRITICAL: A findings
- HIGH: B findings
- MEDIUM: C findings

### Critical Findings (BLOCK)
1. [test_name in file:line] - No assertions found
2. [test_name in file:line] - Tautological assertion: expect(true).toBe(true)

### High Priority Findings (FIX REQUIRED)
1. [test_name in file:line] - Only validates mocks, not behavior
2. [test_name in file:line] - Catch block swallows errors without assertions

### Medium Priority Findings (REVIEW)
1. [file:0] - Happy-path bias: 15 success tests, 0 error tests
2. [test_name in file:line] - Async try/catch missing expect.assertions(n)

### Heuristic Review
- **Assertion Quality**: PASS/WARN/FAIL with examples
- **Coverage Completeness**: PASS/WARN/FAIL with gaps identified
- **Scope Alignment**: PASS/WARN with details

### Test Statistics
- Average assertions per test: 3.2
- Tests without assertions: 2 (8%)
- Async tests without protection: 3 (12%)
- Error test coverage: 20%

### Recommendation
[APPROVED | CHANGES REQUIRED | BLOCKED]

### Action Items
[Specific fixes needed with file:line references]

Finding Categories Reference

no_assertions (CRITICAL)

Test executes code but makes no assertions. Always indicates a problem.

tautological_assertion (CRITICAL)

Assertion always true regardless of implementation (e.g., expect(true).toBe(true)).

vacuous_check (HIGH)

Only checks property exists without validating value (e.g., only .toBeDefined()).

mock_only_validation (HIGH)

Test only verifies mocks were called, doesn't validate actual behavior.

missing_assertion_count (MEDIUM)

Async test with try/catch missing expect.assertions(n) protection.

error_swallowing (HIGH)

Catch block swallows errors without assertions (test always passes).

low_assertion_density (LOW)

Test has suspiciously few assertions for its length (may be superficial).

happy_path_bias (MEDIUM)

File has only success-path tests, no error handling tests.

Integration with QA Protocol

Execute test-standards at Step 3: Change Review (Diff) or Step 7: Tests (Right-Sized) in QA workflow:

1. •Action Agent completes implementation with test changes
2. •Run test-standards script on changed test files
3. •Interpret automated findings
4. •Apply LLM heuristics for semantic review
5. •Generate validation report
6. •If CRITICAL or multiple HIGH findings:
   • •BLOCK validation
   • •Report to Traycer with specific file:line references
   • •Delegate to Action Agent for fixes
   • •Re-run validation after fixes
7. •Continue with remaining QA steps

Quick Reference

For detailed examples and patterns, read references/test-quality-patterns.md:

• •Mesa-optimization anti-patterns with code examples
• •Good vs bad test structures by language
• •Async test protection patterns
• •Happy-path bias examples
• •Error-swallowing anti-patterns

Resources

• •

  Scripts:

  • •scripts/test_quality_scanner.py - Automated test quality detection
• •

  References:

  • •references/test-quality-patterns.md - Detailed patterns and examples

Notes

• •Run script first for deterministic checks (fast, reliable)
• •Use LLM heuristics for semantic/contextual evaluation
• •Always provide file:line references in reports
• •CRITICAL findings must block merge
• •Document justified exceptions in test comments