Terraform Provider Tests

Analyze test coverage and improve Terraform provider acceptance tests using modern patterns from terraform-plugin-testing v1.13.3+.

Summary & Next Steps

When invoked, I will:

1. •Run gap analysis: python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap_$(date +%Y%m%d_%H%M%S).md
2. •Generate timestamped report in ./ai_reports/
3. •Provide succinct summary with:
   • •Overall grade (A/B/C)
   • •Top 3 findings
   • •Recommended next action

Priority levels:

• •P1 (Critical): Missing drift/import tests, heavy legacy usage (>20 calls)
• •P2 (Important): Missing idempotency, moderate legacy (5-20 calls)
• •P3 (Cleanup): Light legacy (<5 calls)

Communication style: Be succinct. Provide summary + single recommended next step.

Quick Start

Common Usage Scenarios

Analyze test modernization gaps:

python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap_$(date +%Y%m%d_%H%M%S).md

Analyze a specific test file: "Analyze resource_example_test.go for coverage gaps"

Verify compilation after changes:

./.claude/skills/terraform-provider-tests/verify_compilation.sh ./internal/provider/

Validate all examples:

export BCM_ENDPOINT="https://..." BCM_USERNAME="..." BCM_PASSWORD="..."
./.claude/skills/terraform-provider-tests/test-examples.sh

Recommended Naming Convention

All gap analysis reports should use the tf_provider_tests_* naming pattern in the ./ai_reports/ directory:

Timestamp format: $(date +%Y%m%d_%H%M%S) generates YYYYMMDD_HHMMSS

Workflow Decision Tree

Starting a modernization project? → Begin with Gap Analysis

Have gap analysis report? → Proceed to Pattern Application

Made code changes? → Run Verification

All changes complete? → Complete Testing

Need to validate examples? → Run Example Testing

Phase 1: Gap Analysis (Automated)

Tool: scripts/analyze_gap.py

Automatically scan test files and generate comprehensive gap analysis report.

Usage

python3 scripts/analyze_gap.py <test_directory> [--output report.md]

Example (recommended naming pattern):

python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap_$(date +%Y%m%d_%H%M%S).md

Simple filename (for one-time analysis):

python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap.md

What It Detects

• •✅ Legacy Check: resource.TestCheckResourceAttr() patterns
• •✅ Missing drift detection tests
• •✅ Missing import tests
• •✅ Missing idempotency checks
• •✅ Modern pattern adoption statistics
• •✅ ID consistency tracking issues
• •✅ Prioritized recommendations

Output

Markdown report with:

• •Codebase statistics (line counts by file type, test-to-impl ratio)
• •Executive summary with overall grade (A/B/C)
• •File-by-file analysis with status
• •Prioritized recommendations (High/Medium/Low)
• •Modern pattern quick reference

Review the report to understand:

• •Which files need the most work
• •What patterns are missing
• •Overall modernization progress

Phase 2: Prioritization

Focus on high-impact changes first:

Priority 1 (Critical) ⚠️

• •Missing drift detection tests
• •Missing import tests
• •Heavy legacy usage (>20 calls per file)

Priority 2 (Important) 📋

• •Missing idempotency checks
• •Moderate legacy usage (5-20 calls)
• •Mixed patterns (legacy + modern)

Priority 3 (Cleanup) 📝

• •Light legacy usage (<5 calls)
• •Documentation improvements

Phase 3: Pattern Application (Guided)

For each file, apply patterns in this order:

Step 1: Add Missing Tests

Missing drift detection? → See references/pattern_templates.md → "Drift Detection Test"

Missing import test? → See references/pattern_templates.md → "Import Test Step"

Missing idempotency checks? → See references/pattern_templates.md → "Idempotency Verification"

Step 2: Convert Legacy to Modern

Replace Check: resource.ComposeAggregateTestCheckFunc() with ConfigStateChecks.

Before:

Check: resource.ComposeAggregateTestCheckFunc(
    resource.TestCheckResourceAttr("example_resource.test", "name", "expected"),
),

After:

ConfigStateChecks: []statecheck.StateCheck{
    statecheck.ExpectKnownValue(
        "example_resource.test",
        tfjsonpath.New("name"),
        knownvalue.StringExact("expected"),
    ),
},

Type mapping → See references/pattern_templates.md

Step 3: Add Required Imports

import (
    "github.com/hashicorp/terraform-plugin-testing/helper/resource"
    "github.com/hashicorp/terraform-plugin-testing/plancheck"
    "github.com/hashicorp/terraform-plugin-testing/statecheck"
    "github.com/hashicorp/terraform-plugin-testing/knownvalue"
    "github.com/hashicorp/terraform-plugin-testing/tfjsonpath"
    "github.com/hashicorp/terraform-plugin-testing/compare"
)

For drift tests, also add: "context", "encoding/json", "time"

Phase 4: Verification (Automated)

Tool: scripts/verify_compilation.sh

Quick compilation check without running tests.

Usage

./.claude/skills/terraform-provider-tests/scripts/verify_compilation.sh <test_directory>

Example:

./.claude/skills/terraform-provider-tests/scripts/verify_compilation.sh ./internal/provider/

What It Validates

• •✅ Go syntax correctness
• •✅ No compilation errors
• •✅ Import completeness
• •✅ Statistics (file count, test count)

If compilation fails:

1. •Review error messages
2. •Check for missing imports
3. •Verify block closures
4. •Re-run after fixes

Phase 5: Testing

Quick Compile Check

go test -c ./internal/provider/ -o /tmp/provider_tests

Parallel Test Execution (Recommended)

Tool: scripts/run_tests_parallel.sh

Run acceptance tests concurrently per file for faster execution.

Usage:

# Run all acceptance tests with 15 concurrent files
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh

# Run only resource tests with higher concurrency
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh --resources-only -c 8

# Run only data source tests
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh --data-sources-only

# Run tests matching specific pattern
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh -p "TestAccCMPartSoftwareImage"

# Run tests from specific file
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh -f resource_cmpart_softwareimage_test.go

# Verbose output with detailed test logs
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh --verbose

Options:

• •-d, --dir DIR - Test directory (default: ./internal/provider)
• •-p, --pattern PATTERN - Test pattern to match (default: TestAcc)
• •-c, --concurrency N - Max concurrent test files (default: 4)
• •-t, --timeout DURATION - Timeout per test file (default: 30m)
• •-f, --file FILE - Run only tests from specific file
• •--resources-only - Run only resource tests
• •--data-sources-only - Run only data source tests
• •--verbose - Show detailed test output
• •--no-color - Disable colored output

Benefits:

• •⚡ Faster execution (4x-8x speedup with proper concurrency)
• •📊 Per-file progress tracking
• •🎯 Aggregated summary with pass/fail counts
• •🔍 Automatic failure highlighting

Single Test (Sequential)

TF_ACC=1 go test -v -timeout 30m ./internal/provider/ -run "^TestAccResource_Specific$"

Full Suite (Sequential)

TF_ACC=1 go test -v -timeout 120m ./internal/provider/

Phase 6: Example Testing (Automated)

Tool: scripts/test-examples.sh

Validate all Terraform examples in the examples/ directory by building the provider, executing examples, and cleaning up test resources.

Usage

# Run all examples (requires BCM credentials)
export BCM_ENDPOINT="https://172.21.15.254:8081"
export BCM_USERNAME="root"
export BCM_PASSWORD="your-password"
./scripts/test-examples.sh

# Quick validation with existing provider build
SKIP_BUILD=true ./scripts/test-examples.sh

# Test only data sources (parallel, ~10s)
./scripts/test-examples.sh --data-sources-only

# Test only resources (sequential, ~19s)
./scripts/test-examples.sh --resources-only

# Debug a failing test
./scripts/test-examples.sh --verbose --no-cleanup

# Cleanup orphaned resources from failed tests
./scripts/test-examples.sh --cleanup-only

What It Validates

• •✅ Examples compile and initialize successfully
• •✅ Provider configuration is correct
• •✅ Schema validation passes
• •✅ Plan generation works (all examples)
• •✅ Apply/destroy cycle succeeds (test-citest examples only)
• •✅ Resources are properly cleaned up

Execution Strategy

Data Sources (parallel):

• •Run up to 4 examples concurrently (configurable with PARALLEL_LIMIT)
• •Fast validation (init → validate → plan)
• •No actual resources created

Resources (sequential):

• •Run one at a time (state-modifying operations)
• •Full lifecycle testing for test-citest examples (init → validate → plan → apply → destroy)
• •Plan-only validation for documentation examples

Test Phases

1. •Environment validation - Verify BCM credentials
2. •Provider build - Compile provider binary (skippable with SKIP_BUILD=true)
3. •Example discovery - Find all .tf files in examples/data-sources/*/ and examples/resources/*/
4. •Example testing - Execute terraform commands on each example
5. •Cleanup - Remove test resources (citest-* prefix) with retry logic

Environment Variables

Required:

• •BCM_ENDPOINT - BCM API endpoint
• •BCM_USERNAME - BCM authentication username
• •BCM_PASSWORD - BCM authentication password

Optional:

• •PROVIDER_VERSION - Provider version (default: 0.1.0)
• •SKIP_BUILD - Skip build phase (default: false)
• •PARALLEL_LIMIT - Max parallel data source tests (default: 4)
• •CLEANUP_RETRIES - Max cleanup retry attempts (default: 4)
• •VERBOSE - Enable verbose logging (default: false)

Exit Codes

• •0 - All tests passed, cleanup successful
• •1 - One or more tests failed
• •2 - Configuration error (missing env vars)
• •3 - Provider build failed
• •130 - Interrupted by user (Ctrl+C)

Best Practices

Example Naming:

• •Use citest- prefix for resources that need cleanup
• •Examples in test-citest/ directories undergo full apply/destroy
• •Other examples are validated with plan-only

Provider Configuration:

• •Examples should NOT include provider blocks
• •Script automatically injects provider config with environment variables
• •Use insecure_skip_verify = true for self-signed certs

Resource Cleanup:

• •Script automatically cleans up resources with citest- prefix
• •Uses exponential backoff retry for cleanup failures
• •Run --cleanup-only to manually cleanup orphaned resources

When to Use Example Testing

Run example testing when:

• •Adding new examples to examples/ directory
• •Modifying provider schema or behavior
• •Before releasing a new provider version
• •Debugging example-specific issues
• •Validating documentation accuracy

Integration with CI/CD

Example testing complements acceptance tests:

Acceptance Tests (make testacc):

• •Full CRUD operation validation
• •Import and drift detection
• •Comprehensive error handling
• •Run on every commit

Example Tests (./scripts/test-examples.sh):

• •Documentation accuracy validation
• •End-user workflow verification
• •Multi-example compatibility
• •Run before releases

Completion Criteria

A fully modernized test file has:

• •✅ Zero legacy Check blocks
• •✅ All state assertions use statecheck.ExpectKnownValue()
• •✅ Idempotency checks after Create and Update
• •✅ Import test with ImportStateVerify
• •✅ Drift detection test (resources only)
• •✅ ID consistency tracking with CompareValue
• •✅ All tests compile and pass
• •✅ All examples validate successfully

ID Consistency Tracking

The analyzer detects inconsistent usage of the .id property across test steps.

Why It Matters

Resource IDs should remain stable across:

• •Initial creation
• •Import operations
• •Update operations

Inconsistent ID handling can indicate:

• •Resource recreation instead of in-place updates
• •Import state mismatches
• •State management bugs

What the Analyzer Detects

Correct Pattern

func TestAccResource_Complete(t *testing.T) {
    // Initialize ID tracker BEFORE Steps
    compareID := statecheck.CompareValue(compare.ValuesSame())

    resource.Test(t, resource.TestCase{
        Steps: []resource.TestStep{
            // Step 1: Create - track ID
            {
                Config: testAccResourceConfig(name),
                ConfigStateChecks: []statecheck.StateCheck{
                    compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")),
                },
            },
            // Step 2: Import - track ID
            {
                ResourceName:      "example_resource.test",
                ImportState:       true,
                ImportStateVerify: true,
                ConfigStateChecks: []statecheck.StateCheck{
                    compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")),
                },
            },
            // Step 3: Update - track ID
            {
                Config: testAccResourceConfig(name, "updated"),
                ConfigStateChecks: []statecheck.StateCheck{
                    compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")),
                },
            },
        },
    })
}

Common Anti-Patterns

Anti-pattern 1: Legacy ID checks

// ❌ Bad - uses legacy pattern
Check: resource.ComposeAggregateTestCheckFunc(
    resource.TestCheckResourceAttrSet("example_resource.test", "id"),
),

Anti-pattern 2: Inconsistent tracking

// ❌ Bad - only tracks ID in Create step, not Import/Update
Steps: []resource.TestStep{
    {
        Config: testAccConfig(name),
        ConfigStateChecks: []statecheck.StateCheck{
            compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")), // ✅
        },
    },
    {
        ResourceName: "example_resource.test",
        ImportState:  true,
        // ❌ Missing: compareID.AddStateValue
    },
    {
        Config: testAccConfig(name, "updated"),
        // ❌ Missing: compareID.AddStateValue
    },
}

Anti-pattern 3: No tracking at all

// ❌ Bad - multiple steps with no ID consistency tracking
compareID := statecheck.CompareValue(compare.ValuesSame()) // Declared but never used!

Common Pitfalls

Missing imports → Add all required imports from Phase 3, Step 3

Wrong knownvalue matcher → Match types correctly: String→StringExact, Bool→Bool, Int64→Int64Exact

Duplicate validation → Remove Check block, keep only ConfigStateChecks

References

This skill includes comprehensive reference documentation:

references/workflow.md

Complete step-by-step modernization workflow with detailed guidance for each phase, common pitfalls, and completion criteria.

When to read: For detailed phase-by-phase instructions.

references/pattern_templates.md

Ready-to-use code templates for all modern patterns:

• •Legacy to modern conversion examples
• •Idempotency verification
• •Import test steps
• •Drift detection tests (complete template)
• •ID consistency tracking
• •knownvalue type matchers
• •Complete test examples

When to read: When applying specific patterns to code.

references/hashicorp_official.md

Consolidated HashiCorp official documentation:

• •TestCase and TestStep structure
• •State checks (statecheck package)
• •Plan checks (plancheck package)
• •Value comparers (compare package)
• •Import mode testing
• •Official testing patterns

When to read: For authoritative information on terraform-plugin-testing features.

Time Estimates

• •Simple file (1-2 tests, <10 legacy checks): 15-30 minutes
• •Medium file (3-5 tests, 10-30 legacy checks): 30-60 minutes
• •Complex file (>5 tests, >30 legacy checks): 1-2 hours

Total project (7 resource + 7 data source files): 4-8 hours across multiple sessions.

Hybrid Approach

This skill uses a hybrid approach:

Automation for:

• •✅ Gap analysis (find legacy patterns, missing tests)
• •✅ Compilation verification (syntax checking)

Guided assistance for:

• •✅ Code changes (apply patterns with context)
• •✅ Complex operations (drift tests, API integration)
• •✅ Decision making (prioritization, test design)

This balance maintains control while automating tedious tasks.