ai-ad-api-automation-test

Note: External URLs are reference-only and may change. The patterns and practices
have been internalized in this skill and AUTOMATION_TEST_SPEC_v1.4.md.

Examples:
- test_create_user__success
- test_invalid_email__raises_validation_error
- test_submit_raw__status_becomes_trend_pending
- test_media_buyer_approve__returns_403
- test_negative_amount__returns_BIZ_100

1. Test<Module>HappyPath
   - Complete successful flow from start to terminal state
   - Reference: STATE_MACHINE.md transitions

2. Test<Module>Permissions
   - Permission boundary tests for each role
   - Reference: AUTH_SPEC.md permission matrix

3. Test<Module>StateMachineViolations
   - Illegal state transition tests
   - Reference: STATE_MACHINE.md whitelist

4. Test<Module>ErrorCodes
   - Error code validation tests
   - Reference: ERROR_CODES_SOT.md definitions

SoT References:
- docs/sot/STATE_MACHINE.md v2.8 Section X.X (specific rule)
- docs/sot/ERROR_CODES_SOT.md v2.2 (error code prefix)
- docs/sot/API_SOT.md v9.4 (API endpoint)
"""
```

SoT References:
- docs/sot/STATE_MACHINE.md v2.8 Section X (state machine)
- docs/sot/ERROR_CODES_SOT.md v2.2 (error codes)
- docs/sot/API_SOT.md v9.4 (API contracts)
- docs/sot/AUTH_SPEC.md v2.1 (permissions)
"""

import pytest
from decimal import Decimal
from datetime import date

from backend.tests.common.factories import create_user, create_<entity>
from backend.tests.common.state_asserts import (
    assert_status_transition,
    assert_valid_transition,
    assert_terminal_state
)
from backend.tests.common.error_helpers import (
    assert_error_response,
    assert_success_response
)
from backend.tests.api.conftest import auth_header


@pytest.mark.api
class Test<Module>HappyPath:
    """
    Happy Path: Complete successful flow

    SoT Reference: STATE_MACHINE.md v2.8 Section X.X
    """

    def test_full_flow__start_to_terminal(self, client, db_session, *_tokens):
        """Complete happy path test"""
        # Arrange
        # Act
        # Assert
        pass


@pytest.mark.api
class Test<Module>Permissions:
    """
    Permission boundary tests

    SoT Reference: AUTH_SPEC.md v2.1 Section 3
    """

    def test_unauthorized_role__returns_403(self, client, db_session, token):
        """Role X cannot perform action Y"""
        pass


@pytest.mark.api
class Test<Module>StateMachineViolations:
    """
    Illegal state transition tests

    SoT Reference: STATE_MACHINE.md v2.8 Section X.X (whitelist)
    """

    def test_illegal_transition__returns_STATE_001(self, client, db_session):
        """<from_state> cannot directly transition to <to_state>"""
        pass


@pytest.mark.api
class Test<Module>ErrorCodes:
    """
    Error code validation tests

    SoT Reference: ERROR_CODES_SOT.md v2.2
    """

    def test_invalid_input__returns_VALIDATION_001(self, client):
        """Missing required field"""
        pass
```

Install: pip install -r backend/requirements-test.txt

Execution Time Limits (per test case, AUTOMATION_TEST_SPEC_v1.4.md 第 2 章):
| Level | Max Time | Timeout Action | Description |
|-------|----------|----------------|-------------|
| L0 Unit | < 100ms | Fail | No external dependencies |
| L1 Integration | < 500ms | Fail | Uses test database |
| L2 API | < 1s | Warn | HTTP interface tests |
| L3 E2E | < 5s | Warn | Complete business flows |

Note: Coverage thresholds and execution time limits are independent metrics.
- Coverage: Measured via pytest-cov, enforced by --cov-fail-under
- Execution time: Measured per test, enforced by pytest-timeout

**Timestamp**: YYYY-MM-DD HH:MM:SS
**Mode**: RUN
**Target**: <module or level>

### Results Summary
| Metric | Value |
|--------|-------|
| Total Tests | N |
| Passed | X |
| Failed | Y |
| Skipped | Z |
| Coverage | XX% |

### Failed Tests
- test_module.py::TestClass::test_function
  - Error: <error message>
  - Location: <file>:<line>

### Coverage by Module
| Module | Coverage |
|--------|----------|
| backend/services/ | XX% |
| backend/api/ | XX% |

### SoT Alignment Check
- [x] STATE_MACHINE.md transitions covered
- [x] ERROR_CODES_SOT.md codes validated
- [ ] Missing: <specific coverage gaps>
```

const newman = require("newman");
const path = require("path");

// Configuration
const config = {
  collection: path.join(__dirname, "../../collections/api_collection.json"),
  environment: path.join(__dirname, "../../environments/local.json"),
  reporters: ["cli", "htmlextra"],
  reporter: {
    htmlextra: {
      export: path.join(__dirname, "../../reports/api_test_report.html"),
      title: "AI Ad Spend API Test Report",
      browserTitle: "API Tests",
      showOnlyFails: false,
      noSyntaxHighlighting: false,
      testPaging: true,
      logs: true,
      omitRequestBodies: false,
      omitResponseBodies: false,
    },
  },
  // Environment variables override
  envVar: [
    { key: "BASE_URL", value: process.env.API_BASE_URL || "http://localhost:8000" },
    { key: "API_VERSION", value: "v1" },
  ],
  // Timeout settings
  timeout: {
    request: 30000,
    script: 30000,
  },
};

// Run Newman
newman.run(config, function (err, summary) {
  if (err) {
    console.error("Newman run failed:", err);
    process.exit(1);
  }

  // Output summary
  console.log("\n=== Test Summary ===");
  console.log(`Total: ${summary.run.stats.tests.total}`);
  console.log(`Passed: ${summary.run.stats.tests.total - summary.run.stats.tests.failed}`);
  console.log(`Failed: ${summary.run.stats.tests.failed}`);

  // Exit with error code if any tests failed
  if (summary.run.stats.tests.failed > 0) {
    process.exit(1);
  }
});
```

**Generated**: YYYY-MM-DD HH:MM:SS
**Project**: AI Ad Spend System
**Baseline**: AUTOMATION_TEST_SPEC_v1.4

## Executive Summary

| Metric | Value | Status |
|--------|-------|--------|
| Total Tests | N | - |
| Pass Rate | XX% | OK/WARN/FAIL |
| Coverage | XX% | OK/WARN/FAIL |
| SoT Alignment | XX% | OK/WARN/FAIL |

## Test Results by Level

### L0 Unit Tests
- Total: N
- Passed: X
- Failed: Y
- Coverage: XX%

### L1 Integration Tests
- Total: N
- Passed: X
- Failed: Y
- Coverage: XX%

### L2 API Tests
- Total: N
- Passed: X
- Failed: Y
- Coverage: XX%

## SoT Alignment Matrix

| SoT Document | Coverage | Missing |
|--------------|----------|---------|
| STATE_MACHINE.md | XX/YY transitions | [list] |
| ERROR_CODES_SOT.md | XX/YY codes | [list] |
| API_SOT.md | XX/YY endpoints | [list] |
| AUTH_SPEC.md | XX/YY permissions | [list] |

## Failed Test Details

### test_module::test_function
- **Status**: FAILED
- **Error**: <error message>
- **Location**: <file>:<line>
- **SoT Reference**: <relevant SoT>
- **Suggested Fix**: <recommendation>

## Coverage Gaps

1. **Missing Happy Path**: <module> lacks complete flow test
2. **Missing Permission Test**: <role> + <action> combination
3. **Missing Error Code**: <code> not validated

## Recommendations

1. [ ] Add test for <specific scenario>
2. [ ] Increase coverage for <module>
3. [ ] Update test to align with SoT v<new_version>
```

<module name="topup_request">
  <router>backend/routers/topup.py</router>
  <service>backend/services/topup_service.py</service>
  <sot_documents>
    - docs/sot/TOPUP_SOT.md v1.0
    - docs/sot/STATE_MACHINE.md v2.8 Section 9
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/API_SOT.md v9.4
  </sot_documents>
  <state_machine>STATE_MACHINE.md v2.8 Section 9 (7-state: draft → pending_review → finance_approve → paid → completed)</state_machine>
  <test_path>backend/tests/api/test_topup_request_flow_generated.py</test_path>
</module>

<module name="reconciliation">
  <router>backend/routers/reconciliation.py</router>
  <service>backend/services/reconciliation_service.py</service>
  <sot_documents>
    - docs/sot/RECONCILIATION_SOT.md v1.0
    - docs/sot/STATE_MACHINE.md v2.8 Section 11
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/API_SOT.md v9.4
  </sot_documents>
  <state_machine>STATE_MACHINE.md v2.8 Section 11 (5-state: draft → pending_review → approved/needs_adjustment → completed)</state_machine>
  <test_path>backend/tests/api/test_reconciliation_flow_generated.py</test_path>
</module>

<module name="ledger">
  <router>backend/routers/ledger.py</router>
  <service>backend/services/ledger_service.py</service>
  <sot_documents>
    - docs/sot/DATA_SCHEMA.md v5.11 §3.4.4
    - docs/sot/DATA_SCHEMA.md v5.6
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/API_SOT.md v9.4
  </sot_documents>
  <state_machine>N/A (no state machine, ledger entries are immutable records)</state_machine>
  <test_path>backend/tests/api/test_ledger_flow_generated.py</test_path>
</module>

<module name="ad_accounts">
  <router>backend/routers/ad_accounts.py</router>
  <service>backend/services/ad_account_service.py</service>
  <sot_documents>
    - docs/sot/API_SOT.md v9.4 Section 7
    - docs/sot/STATE_MACHINE.md v2.8 Section 7.1
    - docs/sot/DATA_SCHEMA.md v5.6
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/AUTH_SPEC.md v2.1
  </sot_documents>
  <state_machine>STATE_MACHINE.md v2.8 Section 7.1 (ad_accounts.status: new → testing → active → suspended → dead → archived)</state_machine>
  <test_path>backend/tests/api/test_ad_accounts_flow_generated.py</test_path>
</module>

<module name="suppliers">
  <router>backend/routers/suppliers.py</router>
  <service>backend/services/supplier_service.py</service>
  <sot_documents>
    - docs/sot/API_SOT.md v9.4
    - docs/sot/DATA_SCHEMA.md v5.6
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/AUTH_SPEC.md v2.1
    - docs/sot/DATA_SCHEMA.md v5.11 §3.4.4 (SUPPLIER ledger)
  </sot_documents>
  <state_machine>N/A (no state machine, suppliers.status is simple active/inactive)</state_machine>
  <test_path>backend/tests/api/test_suppliers_flow_generated.py</test_path>
</module>

<module name="settlements">
  <router>backend/routers/settlements.py</router>
  <service>backend/services/settlement_service.py</service>
  <sot_documents>
    - docs/sot/API_SOT.md v9.4
    - docs/sot/DATA_SCHEMA.md v5.11 §3.4.4 (ledger integration)
    - docs/sot/DATA_SCHEMA.md v5.6
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/AUTH_SPEC.md v2.1
  </sot_documents>
  <state_machine>N/A (no state machine, settlements may have approval workflow but not defined in STATE_MACHINE.md)</state_machine>
  <test_path>backend/tests/api/test_settlements_flow_generated.py</test_path>
</module>

<module name="projects">
  <router>backend/routers/projects.py</router>
  <service>backend/services/project_service.py</service>
  <sot_documents>
    - docs/sot/API_SOT.md v9.4 Section 6
    - docs/sot/STATE_MACHINE.md v2.8 Section 5
    - docs/sot/DATA_SCHEMA.md v5.6
    - docs/sot/ERROR_CODES_SOT.md v2.2
    - docs/sot/AUTH_SPEC.md v2.1
  </sot_documents>
  <state_machine>STATE_MACHINE.md v2.8 Section 5 (projects.status: draft → active → suspended → archived)</state_machine>
  <test_path>backend/tests/api/test_projects_flow_generated.py</test_path>
</module>

<module name="auth">
  <router>backend/routers/authentication.py</router>
  <service>backend/services/supabase_auth_service.py</service>
  <sot_documents>
    - docs/sot/AUTH_SPEC.md v2.1
    - docs/sot/API_SOT.md v9.4
    - docs/sot/ERROR_CODES_SOT.md v2.2
  </sot_documents>
  <state_machine>N/A (no state machine, authentication is stateless)</state_machine>
  <test_path>backend/tests/api/test_auth_flow_generated.py</test_path>
</module>

Available factories (all use keyword-only arguments):
- create_test_project(*, name, owner_id, status, **kwargs) → Dict
- create_test_channel(*, name, channel_type, **kwargs) → Dict
- create_test_ad_account(*, project_id, channel_id, account_name, platform_account_id, balance, **kwargs) → Dict
- create_test_daily_report(*, ad_account_id, report_date, status, spend, impressions, clicks, conversions, **kwargs) → Dict
- create_test_topup_request(*, ad_account_id, amount, status, requested_by, **kwargs) → Dict

Usage:
```python
from backend.tests.common.factories import create_test_daily_report
report = create_test_daily_report(status="raw_submitted", spend=Decimal("100.00"))
```

MUST use factories instead of ad-hoc data creation.

Available assertions:
- assert_daily_report_state(entity, expected_state, msg=None) - DailyReport 8 状态机
- assert_topup_state(entity, expected_state, msg=None) - Topup 状态机
- assert_reconciliation_state(entity, expected_state, msg=None) - Reconciliation 状态机
- assert_state_transition_valid(entity_type, from_state, to_state, msg=None) - 验证状态转换合法性

State whitelist constants:
- DAILY_REPORT_STATES, DAILY_REPORT_TRANSITIONS, DAILY_REPORT_TERMINAL_STATES
- TOPUP_STATES, TOPUP_TRANSITIONS, TOPUP_TERMINAL_STATES
- RECONCILIATION_STATES, RECONCILIATION_TRANSITIONS, RECONCILIATION_TERMINAL_STATES

Usage:
```python
from backend.tests.common.state_asserts import (
    assert_daily_report_state,
    assert_state_transition_valid,
    DAILY_REPORT_TRANSITIONS
)
assert_daily_report_state(report, "trend_pending")
assert_state_transition_valid("daily_report", "raw_submitted", "trend_pending")
```

Available helpers:
- assert_error_code(response, expected_code, msg=None) - 断言特定错误码
- assert_validation_error(response, expected_code=None, msg=None) - 断言 VAL-xxx 错误
- assert_auth_error(response, expected_code=None, msg=None) - 断言 AUTH-xxx 错误
- assert_business_error(response, expected_code=None, msg=None) - 断言 BIZ-xxx 错误

Error code constants:
- ERROR_CODE_PREFIXES: {"VAL", "AUTH", "PERM", "BIZ", "SYS", "DATA", "EXT", "LED", "RPT", "TOP", "REC"}
- VALIDATION_ERROR_CODES: {"VAL-001", "VAL-002", "VAL-003", "VAL-004", "VAL-005"}
- AUTH_ERROR_CODES: {"AUTH-001", "AUTH-002", "AUTH-003", "AUTH-004", "AUTH-005"}
- BUSINESS_ERROR_CODES: {"BIZ-001", "BIZ-002", "BIZ-003", "BIZ-004", "BIZ-005"}

Usage:
```python
from backend.tests.common.error_helpers import assert_error_code, assert_validation_error
assert_error_code(response, "VAL-001")
assert_validation_error(response)  # Any VAL-xxx error
```

This skill operates in READ-ONLY mode for all SoT documents:
- ✅ MAY read SoT documents for reference and validation
- ✅ MAY generate test code that references SoT documents
- ✅ MAY validate test results against SoT specifications
- ❌ MUST NOT modify any files in docs/sot/*
- ❌ MUST NOT propose changes to SoT documents
- ❌ MUST NOT add/remove/rename SoT definitions

If SoT inconsistencies are detected:
1. Log the inconsistency with exact SoT reference
2. Report to user for manual SoT governance process
3. Continue test execution with current SoT values

Example 1b: Generate L2 API tests for Ad Accounts module

Example 1c: Generate L2 API tests for Suppliers module

Example 2: Run all L2 API tests

Example 3: Execute Newman with Postman collection

Example 4: Generate comprehensive test report

</usage>


<!-- ======================================================
   11. Error Handling
====================================================== -->
<error_handling>
Scenario 1: Target module not found
- Report: "Error: Module '<name>' not found in backend/"
- Suggest: List available modules from valid values enumeration
- Action: Abort operation

Scenario 2: SoT document not found
- Report: "Warning: SoT document '<path>' not found"
- Fallback: Continue with available SoT references
- Action: Log warning, proceed with caution

Scenario 3: Test execution fails
- Report: Detailed error with stack trace
- Include: Failed assertion details, SoT reference
- Suggest: Potential fixes based on error type
- Action: Mark test as FAILED, continue suite

Scenario 4: Newman collection invalid
- Report: "Error: Invalid Postman collection format"
- Suggest: Validate JSON structure against Postman schema
- Action: Abort Newman execution

Scenario 5: Coverage below threshold
- Report: "Warning: Coverage XX% below threshold YY%"
- List: Uncovered modules/functions
- Action: Mark build as FAILED if CI blocking

Scenario 6: Output directory not writable
- Report: "Error: Cannot write to '<output_dir>'"
- Suggest: Check directory permissions, use alternative path
- Action: Abort operation

Scenario 7: File already exists (overwrite=false)
- Report: "Warning: File '<path>' already exists"
- Suggest: Set overwrite=true or use different output_dir
- Action: Skip file, continue with next

Scenario 8: Newman/Node.js not available
- Report: "Error: Newman CLI not found"
- Suggest: Run 'npm install -g newman newman-reporter-htmlextra'
- Action: Abort NEWMAN mode

Scenario 9: Execution timeout exceeded
- Report: "Error: Test '<name>' exceeded timeout limit"
- Include: Test level, configured timeout, actual duration
- Action: Mark test as TIMEOUT, continue suite

Scenario 10: Environment variables missing
- Report: "Error: Required environment variable '<var>' not set"
- Suggest: Check environments/*.json or set via shell
- Action: Abort operation
</error_handling>


<!-- ======================================================
   12. Version Notes
====================================================== -->
<VERSION_NOTES>

### v1.3 (2025-12-01)

**Module Mapping Enhancement - Status: beta**

P0 Enhancements:
- Added complete module mapping configuration section (Section 9)
- Extended target_module valid values from 5 to 9 modules:
  - Added: ad_accounts, suppliers, settlements, projects
  - Existing: daily_report, topup_request, reconciliation, ledger, auth
- Added comprehensive module definitions table with router/service/SoT/test paths
- Added detailed module_details XML blocks for each of 9 modules

Module Coverage:
- All 9 modules now have complete configuration:
  - Router file paths (backend/routers/*.py)
  - Service file paths (backend/services/*_service.py)
  - SoT document references (docs/sot/*.md)
  - State machine references (STATE_MACHINE.md sections)
  - Test output paths (backend/tests/api/test_*_flow_generated.py)

Configuration Alignment:
- All module names consistent across:
  - target_module valid values enumeration
  - supported_modules table
  - module_details XML blocks
  - Usage examples

---

### v1.2 (2025-12-01)

**P0/P1 Pre-Launch Remediation - Status: beta**

P0 Fixes:
- Fixed version inconsistency (YAML/XML both v1.2)
- Added Safety Constraints section with SoT read-only declaration
- Fixed coverage_thresholds to include execution time limits
- Aligned thresholds with AUTOMATION_TEST_SPEC_v1.4.md 第 2 章

P1 Fixes:
- Added target_module valid values enumeration
- Added dry_run, output_dir, overwrite parameters
- Added pytest prerequisites/dependencies section
- Updated common_utilities to match actual implementations:
  - factories.py: create_test_* functions
  - state_asserts.py: assert_*_state, assert_state_transition_valid
  - error_helpers.py: assert_error_code, assert_*_error
- Extended error_handling from 5 to 10 scenarios

---

### v1.1 (2025-12-01)

**P0/P1/P2 Audit Fixes - Status: beta**

Infrastructure Created:
- Created backend/tests/common/ directory with utility files:
  - factories.py: Test data factory functions
  - state_asserts.py: State machine assertion helpers
  - error_helpers.py: Error code validation helpers
- Created test layer directories:
  - backend/tests/unit/ (L0)
  - backend/tests/integration/ (L1)
  - backend/tests/api/ (L2)
  - backend/tests/e2e/ (L3)
- Created Newman directories:
  - scripts/newman/
  - collections/
  - environments/

Configuration Updates:
- Updated pytest.ini with e2e and smoke markers
- Created .github/workflows/test-coverage.yml for CI
- Added coverage thresholds per test level

Status Changes:
- Changed status from ready_for_production to beta
- Added Status column to external_references table
- Added external URL availability note

---

### v1.0 (2025-12-01)

**Initial Release**
- Created skill based on sanbercode-api-automation-boilerplate patterns
- Integrated with AUTOMATION_TEST_SPEC_v1.4.md standards
- Support for 4 modes: GENERATE, RUN, NEWMAN, REPORT
- pytest test generation with SoT alignment
- Newman/Postman collection integration for contract testing
- HTML report generation via newman-reporter-htmlextra
- Common utilities integration (factories, state_asserts, error_helpers)

**Reference Projects**
- https://github.com/jfrelis/sanbercode-api-automation-boilerplate
- Newman v6.0.0+
- newman-reporter-htmlextra v1.22.11+

**Alignment**
- MASTER.md v4.6
- SoT Freeze v2.8
- AUTOMATION_TEST_SPEC_v1.4.md

</VERSION_NOTES>

</skill>