Evidence-Based Verification

Core Principle

Claims of success MUST be backed by actual evidence. "Should work" is prohibited.

This skill enforces rigorous verification practices because AI-generated code often appears correct but fails on edge cases, integration points, or real-world data. Every claim must be proven with executed commands and observed results.

When to Use

Apply during Phase 4.5 (Verification) when:

• •Implementation is marked complete
• •Tests "should" be passing
• •Feature appears to work
• •Code changes are ready for review
• •Merge/deployment is being considered

The Iron Law of Verification

Never claim success without running the actual verification command.

Every verification claim requires:

1. •Identify the exact proof command
2. •Run it completely (not partially)
3. •Read full output + exit codes
4. •Verify output supports the claim
5. •Only then state the result

Prohibited Language

These phrases are BANNED in verification:

• •❌ "should work"
• •❌ "seems fine"
• •❌ "appears correct"
• •❌ "looks good"
• •❌ "probably passes"
• •❌ "likely works"
• •❌ "ought to pass"

Replace with evidence-based language:

• •✅ "Tests pass (exit code 0, 42/42 tests green)"
• •✅ "Verified by running: npm test - all assertions passed"
• •✅ "Evidence: coverage report shows 87% (above 80% target)"

Verification Checklist

Run through this checklist for every completed task:

1. Unit Tests

Command: Execute full unit test suite

# Examples by language
npm test                    # JavaScript/TypeScript
pytest                      # Python
go test ./...               # Go
cargo test                  # Rust
mvn test                    # Java
bundle exec rspec           # Ruby

Evidence Required:

• •Exit code is 0
• •All tests passed (X/X green)
• •No skipped tests (or justify why)
• •Duration reasonable (not timing out)

2. Test Coverage

Command: Generate coverage report

# Examples
npm run test:coverage       # JavaScript
pytest --cov=src --cov-report=term
go test -cover ./...
cargo tarpaulin

Evidence Required:

• •Coverage percentage meets target (≥80% by default)
• •New code is covered (not just overall percentage)
• •Critical paths at 100% coverage

3. Integration Tests

Command: Run integration test suite (if applicable)

npm run test:integration
pytest tests/integration/
go test -tags=integration ./...

Evidence Required:

• •All integration tests pass
• •External dependencies properly mocked/stubbed
• •API contracts verified

4. Linting and Style

Command: Run linter

npm run lint                # ESLint
pylint src/
golangci-lint run
cargo clippy

Evidence Required:

• •Zero errors
• •Zero warnings (or justify allowed warnings)
• •Style rules enforced

5. Type Checking

Command: Run type checker (if typed language)

npx tsc --noEmit           # TypeScript
mypy src/                  # Python
# Go/Rust compile-time checked

Evidence Required:

• •No type errors
• •No implicit any (TypeScript)
• •Type safety verified

6. Build Verification

Command: Run production build

npm run build
python setup.py bdist_wheel
go build ./...
cargo build --release

Evidence Required:

• •Build succeeds (exit code 0)
• •No warnings (or document allowed ones)
• •Artifacts generated correctly

7. Functional Verification

Command: Manual execution/smoke test

# Start application
npm start
# Execute specific feature
curl http://localhost:3000/api/endpoint
# Or manual UI testing

Evidence Required:

• •Feature works as specified
• •Edge cases handled
• •Error messages appropriate
• •Performance acceptable

Verification Levels

Match verification rigor to project maturity:

Prototype Projects

Required:

• •Unit tests for new code pass
• •Feature demonstrably works
• •No critical errors

Optional:

• •Coverage targets
• •Integration tests
• •Full lint compliance

Development Projects

Required:

• •All unit tests pass
• •Coverage ≥80%
• •Feature works as specified
• •Linting clean
• •Type checking passes

Optional:

• •Integration tests (recommended)
• •Performance benchmarks

Production Projects

Required:

• •All unit tests pass
• •All integration tests pass
• •Coverage ≥80%
• •All linting rules pass
• •Type checking passes
• •Build succeeds
• •Feature verified manually
• •Performance acceptable

Critical:

• •Security scan clean
• •No known vulnerabilities
• •Breaking changes documented

Regulated Projects

Required: All production checks PLUS

• •Coverage ≥95%
• •Full audit trail
• •Compliance checks pass
• •Documentation complete
• •Human approval obtained

Evidence Documentation

Document verification results clearly:

Good Example

## Verification Results

✅ Unit Tests: PASS
- Command: `npm test`
- Result: 47/47 tests passed
- Duration: 3.2s
- Exit code: 0

✅ Coverage: PASS
- Command: `npm run test:coverage`
- Result: 84% lines, 82% branches
- Target: 80%
- Evidence: coverage/index.html generated

✅ Linting: PASS
- Command: `npm run lint`
- Result: 0 errors, 0 warnings
- Exit code: 0

✅ Build: PASS
- Command: `npm run build`
- Result: dist/ generated successfully
- Size: 127KB (within budget)

✅ Manual Verification: PASS
- Tested user login flow
- Successful authentication
- Error handling verified
- Screenshot: verification/login-success.png

Bad Example

## Verification Results

Tests should pass because I wrote them correctly.
The code looks good and follows best practices.
Linting seems fine, no obvious errors.

Partial Verification is Not Verification

Do not claim:

• •"95% of tests pass" → 5% failing means NOT VERIFIED
• •"Most linting rules pass" → Any error means NOT VERIFIED
• •"Works on my machine" → Must work in CI environment
• •"Manual testing shows it works" → Need automated proof

Full verification required:

• •100% of tests pass (or blockers documented)
• •Zero linting errors (or exceptions justified)
• •Works in all required environments
• •Both automated and manual checks pass

Handling Verification Failures

First Failure

1. •Read the full error message
2. •Identify root cause
3. •Fix the issue
4. •Re-verify completely

Second Failure

1. •Question the test (is it correct?)
2. •Question the specification (was it clear?)
3. •Try alternative implementation
4. •Re-verify completely

Third Failure

STOP. Escalate.

• •Suspect architecture problem, not implementation
• •Review design decisions
• •Ask for human guidance
• •Document the blocker

Alternative Verification Methods

When traditional tests are impractical:

Legacy Code (No Tests)

✅ Acceptable:

• •Write characterization test for changed code
• •Golden file comparison (snapshot)
• •Manual verification checklist with screenshots
• •Document what was verified manually

❌ Not Acceptable:

• •Skip verification entirely
• •Trust that nothing broke
• •"Looks the same" without proof

UI/Visual Changes

✅ Acceptable:

• •Visual regression tests (Percy, Chromatic)
• •Screenshot comparison with before/after
• •Manual checklist with documented steps
• •Storybook component verification

❌ Not Acceptable:

• •"UI looks fine to me"
• •No before/after comparison
• •No evidence captured

Infrastructure Changes

✅ Acceptable:

• •Dry-run/plan preview
• •Canary deployment verification
• •Rollback test execution
• •Drift detection reports

❌ Not Acceptable:

• •"Should work in production"
• •Untested deploy scripts
• •No rollback plan

ML/AI Models

✅ Acceptable:

• •Benchmark on test dataset
• •Regression tests on known inputs
• •Performance metrics comparison
• •A/B test results

❌ Not Acceptable:

• •"Seems to work well"
• •No quantitative metrics
• •Cherry-picked examples

Integration with Workflow

Phase 4.5: Verification (THIS SKILL)

• •Apply evidence-based verification
• •Document all verification results
• •Block progression on failures
• •No "should work" claims allowed

Phase 4: Implementation

• •TDD skill ensures tests written first
• •Tests must pass before marking complete
• •This skill provides verification proof

Phase 5: Review

• •Verification results inform review
• •Evidence of correctness demonstrated
• •Review focuses on quality, not correctness

Verification Commands by Language

JavaScript/TypeScript

# Tests
npm test
npm run test:watch

# Coverage
npm run test:coverage
npx jest --coverage

# Linting
npm run lint
npx eslint .

# Type checking
npx tsc --noEmit

# Build
npm run build

Python

# Tests
pytest
python -m pytest

# Coverage
pytest --cov=src --cov-report=html
coverage run -m pytest

# Linting
pylint src/
flake8 src/
ruff check .

# Type checking
mypy src/

# Build
python -m build

Go

# Tests
go test ./...
go test -v ./...

# Coverage
go test -cover ./...
go test -coverprofile=coverage.out ./...

# Linting
golangci-lint run
go vet ./...

# Build
go build ./...

Rust

# Tests
cargo test
cargo test --all-features

# Coverage
cargo tarpaulin --out Html

# Linting
cargo clippy
cargo clippy -- -D warnings

# Build
cargo build --release

Common Verification Failures

False Positives

Problem: Tests pass but feature doesn't work Cause: Tests don't actually test the requirement Solution: Review test validity, add missing assertions

Environment Differences

Problem: Passes locally, fails in CI Cause: Different dependencies, env vars, or configs Solution: Run tests in clean environment, match CI setup

Flaky Tests

Problem: Tests pass sometimes, fail randomly Cause: Timing issues, shared state, async problems Solution: Fix root cause before proceeding, never ignore

Coverage False Sense

Problem: High coverage but bugs exist Cause: Lines covered but assertions weak Solution: Review test quality, not just quantity

Verification Anti-Patterns

Trust Without Testing

❌ Anti-pattern:

"I ran the tests earlier and they passed,
so the current code should be fine."

✅ Correct:

"Running tests now to verify..."
[execute tests]
"Tests pass: 47/47 green, exit code 0"

Partial Test Runs

❌ Anti-pattern:

"I tested the specific function I changed,
that should be enough."

✅ Correct:

"Running full test suite to catch regressions..."
[execute all tests]
"Full suite passes, including integration tests"

Unverified Agent Claims

❌ Anti-pattern:

"The subagent reported that tests pass."

✅ Correct:

"Verifying subagent's claim by running tests..."
[execute tests directly]
"Confirmed: tests pass as reported"

Skipping Due to Fatigue

❌ Anti-pattern:

"We've been working on this for a while,
let's skip verification and move on."

✅ Correct:

"Long session requires careful verification.
Taking time to verify properly..."
[complete full verification]
"Verified before proceeding"

Checklist Before Declaring Complete

Before saying "implementation is complete":

• • Unit tests executed and pass (100%)
• • Coverage measured and meets target
• • Integration tests pass (if applicable)
• • Linting passes with zero errors
• • Type checking passes (if applicable)
• • Build succeeds
• • Manual verification performed
• • Performance acceptable
• • Security considerations checked
• • All evidence documented
• • No "should work" language used

If ANY item unchecked → implementation NOT complete.

Summary

Remember:

• •Evidence over assumptions
• •Run actual commands, read actual output
• •Never trust without verification
• •Document all verification results
• •Partial success is not success
• •When verification fails repeatedly, escalate
• •No "should work" - only "verified to work"

Verification is not optional. It's the gate between code and confidence.