Test Strategy (Foundational Skill)

You are the foundational test strategy authority. All other skills—architect, coder, reviewer—MUST consult you before making decisions about testing.

The Overarching Question

What evidence do I need to convince the user that my code correctly implements the specification?

Every test exists to answer this question. Tests are not bureaucracy—they are evidence that your code works. If a test doesn't provide evidence, delete it.

Tests Serve Three Purposes

The Cardinal Rule: No Mocking

Mocking is always wrong. There is no exception.

Mocking gives you a test that passes while your production code fails. This is worse than no test at all.

If you feel you need to mock:

1. •Redesign using dependency injection with real in-memory implementations, OR
2. •Test at a different level—push to Level 2 or 3 where real dependencies are available

The Three Levels

┌─────────────────────┐
│      LEVEL 3        │  "Does it work in the real world?"
│   System / E2E      │  Real credentials, real services
│                     │  Full user workflows
└──────────┬──────────┘
           │
┌──────────▼──────────┐
│      LEVEL 2        │  "Does it work with real infrastructure?"
│    Integration      │  Real binaries, real databases
│                     │  Test harnesses required
└──────────┬──────────┘
           │
┌──────────▼──────────┐
│      LEVEL 1        │  "Is our logic correct?"
│    Unit / Pure      │  Standard dev environment only
│                     │  DI with real in-memory implementations
└─────────────────────┘

Level 1: Standard Dev Environment

The question: Is our logic correct, independent of any external system?

Allowed:

Forbidden:

Critical filesystem rule: All Level 1 tests MUST use OS-provided temporary directories exclusively. Never write outside temp directories.

Level 2: Project-Specific Dependencies

The question: Does our code correctly interact with real external dependencies?

Covers:

• •Project-specific tools: Hugo, Caddy, FFmpeg, custom binaries
• •Project-specific build tools: Make, Gradle, Maven
• •Containerized services: Docker databases, message queues

Required before writing: Document the test harness for each dependency.

If you don't know the harness, STOP and ask:

I need to write integration tests for [dependency].

To proceed, I need to know:

1. •What test harness exists or should I build?
2. •How do I start/stop/reset it?
3. •Where are fixture files or seed data?
4. •What environment variables configure it?

Level 3: Real Environment

The question: Does the complete system work the way users will actually use it?

Covers:

• •Real credentials against real (test) environments
• •Third-party services in production or staging
• •Browser-based testing for web applications
• •Complete user workflows end-to-end

Required before writing: Document credentials and test accounts.

If you don't know the credentials, STOP and ask:

I need to write end-to-end tests that use [external service].

To proceed, I need to know:

1. •Where are the test credentials stored?
2. •What test accounts/environments exist?
3. •Are there rate limits or quotas?
4. •How do I reset test data between runs?

The Three Dimensions

Every test level offers different tradeoffs across three orthogonal dimensions:

1. Detection: What Bugs Can This Level Find?

Each level has a different lens—not progressively broader, but genuinely different:

Key insight: L1 property-based tests catch algorithmic bugs that L3 almost never detects. L2 randomized harnesses catch race conditions that L1 can't see.

2. Validity: What Does a Passing Test Prove?

Key insight: "E2E tests pass" does NOT mean "it works for users." L3 has the highest false confidence risk.

3. Cost: What Investment Does This Level Require?

Key insight: A test cheap to write can be expensive to maintain. L3 tests often become "tests you're afraid to touch."

Where Evidence Lives

Some outcomes can only be proven at specific levels:

The Clipboard Example

A "copy to clipboard" React component:

• •L1 test (component renders): Proves nothing about clipboard functionality
• •L3 test (actually copies in browser): Proves the feature works

Best combination: Skip L1/L2 entirely, write L3 tests in target browsers.

The Pricing Engine Example

A complex pricing calculation with discounts, taxes, promotions:

• •L2 only (integration test checks total): Knows IF wrong, not WHERE the bug is
• •L1 + L2: L1 isolates which rule is broken; L2 confirms end-to-end

Best combination: L1 with full 4-part progression + L2 for integration.

Add Lower Levels for Debuggability

When Level 2 or 3 is required for evidence, add Level 1 tests ONLY if:

1. •The code is complex—your logic (algorithms, parsers, rules), not library wiring
2. •Debugging will be hard—when the higher-level test fails, will you know where to look?
3. •Property-based testing adds value—would generated inputs find edge cases?

Trust the Library

Libraries like argparse, Zod, pydantic, js-yaml are battle-tested. Don't test that they work—test YOUR logic.

Don't test library behavior:

# BAD: Tests argparse, not your code
def test_verbose_flag_is_parsed():
    args = parser.parse_args(["--verbose"])
    assert args.verbose is True

Do test your behavior:

# GOOD: Tests your logic that uses the parsed result
def test_verbose_mode_produces_detailed_output():
    output = run_command(verbose=True)
    assert "DEBUG:" in output

Randomized Test Harnesses

Always ask: What is the data structure that describes the fixture?

Example: When testing directory tree operations:

1. •The underlying structure is a DAG (directed acyclic graph)
2. •Generate a DAG data structure first
3. •Test your logic against the DAG at Level 1
4. •Convert to actual directories in temp directory for Level 2

Seeding

• •Seeds should derive from system time (different every run)
• •Show seed on failure for reproduction
• •This maximizes variety while enabling reproduction

The 4-Part Progression

Organize tests at any level to serve all three purposes (code/evidence/debug):

Part 0: Shared Test Values

Create a test values file with named, typed data:

export const TYPICAL = {
  BASIC: { input: "simple", expected: 42 },
  COMPLEX: { input: "with-flags", expected: 100 },
} as const;

export const EDGES = {
  EMPTY: { input: "", expected: 0 },
  MAX: { input: "x".repeat(1000), expected: "ERROR" },
} as const;

Part 1: Named Typical Cases

One it() per category. When test fails, you know EXACTLY which case.

Part 2: Named Edge Cases

One it() per boundary condition. Each boundary is independently debuggable.

Part 3: Systematic Coverage Loop

Loop over all known cases. Should ONLY fail if Parts 1-2 missed a category.

Part 4: Generated/Property-Based

Reproducible via seed. Escalate from debuggable loops to comprehensive properties.

Level Breadth

Test Location in CODE Framework

Tests are co-located with specs in the spx/ tree:

The invariant: Tests listed in outcomes.yaml MUST ALWAYS PASS (precommit validates this).

No graduation: Tests stay where they are. The outcomes.yaml file tracks which tests have passed. Test level is indicated by filename suffix:

• •*.unit.test.{ts,py} - Level 1 (Vitest/pytest)
• •*.integration.test.{ts,py} - Level 2 (Vitest/pytest)
• •*.e2e.test.{ts,py} - Level 3, non-browser (Vitest/pytest)
• •*.e2e.spec.{ts,py} - Level 3, browser-based (Playwright)

Runner separation: Vitest/pytest find *.test.* files, Playwright finds *.spec.* files. No config needed.

Stories persist as containers in the tree. Completion is tracked by outcomes.yaml, not by moving files.

Test Infrastructure

Keep test infrastructure separate from tests. Categories:

1. Test Environment Context Managers

Shared utilities (like withTestEnv) that handle:

• •Seeding and reproducibility
• •Temp directory lifecycle
• •Environment variable isolation
• •Shared setup/teardown

2. Containerized Services

Local databases, dev servers, message queues. Managed via docker-compose.

3. Fixtures

Named test values (TYPICAL, EDGES)—static data collections.

4. Generators

Randomized data generation with seeding for reproducibility.

Dependency Injection Pattern

# BAD: Hardcoded dependency, requires mocking
class OrderProcessor:
    def process(self, order):
        db = PostgresDatabase()  # Hardcoded!
        db.save(order)


# GOOD: Injected dependencies, testable without mocks
class OrderProcessor:
    def __init__(self, repository):
        self.repository = repository

    def process(self, order):
        self.repository.save(order)


# Level 1 test: Real in-memory implementation
def test_order_processing_saves():
    saved = []

    class InMemoryRepo:
        def save(self, order):
            saved.append(order)

    processor = OrderProcessor(InMemoryRepo())
    processor.process(Order(customer="alice"))

    assert len(saved) == 1

Quick Reference: Level Selection

Checklist Before Declaring Tests Complete

• • Evidence exists at the level where it can be proven
• • No mocking anywhere—DI with real implementations
• • Level 2 harnesses documented
• • Level 3 credentials documented (not hardcoded)
• • Tests verify behavior, not implementation
• • Regression tests all pass

When You're Stuck

For Level 1:

Can I verify this behavior using only the test runner, language primitives, temp dirs, and DI?

If no → move to Level 2

For Level 2:

What test harness do I need?

If you don't know → STOP AND ASK THE USER

For Level 3:

Where are the credentials?

If you don't know → STOP AND ASK THE USER

The goal is not "passing tests" or "high coverage"—it's justified confidence that your code works in the real world.