CI Pipeline Skill

This skill runs the full CI pipeline with detailed feedback, auto-fixing lint issues, and reviewing test coverage.

Arguments

• •--eval - Include make test-eval (runs Claude tool call evaluation tests, takes a long time)
• •--skip-docs - Skip documentation validation
• •--coverage-only - Only run test coverage analysis

Pipeline Steps

Execute each step in order. Stop on failure unless the step is auto-fixable.

Step 1: Install Dependencies

poetry install

Verify the virtual environment is set up correctly.

Step 2: Format Code

poetry run ruff format src/ tests/

This auto-formats all code. Report what files were changed (if any).

Step 3: Lint with Auto-Fix

poetry run ruff check --fix src/ tests/

If there are fixable issues, they will be auto-fixed. After auto-fix, run again without --fix to see remaining issues:

poetry run ruff check src/ tests/

If there are remaining unfixable issues, report them and stop.

Step 4: Type Checking

poetry run mypy --warn-unreachable --warn-redundant-casts --warn-unused-ignores src/

Report any type errors. Stop if there are errors.

Step 5: Run Unit Tests with Coverage

poetry run pytest tests/unit/ -v --cov=honeycomb --cov-report=term-missing --cov-branch

After tests complete:

1. •Check overall coverage percentage - current baseline is ~38% (CLI and integration code is tested via live tests)
2. •Identify uncovered lines - focus on models and validation code (should be 80%+)
3. •Check for new functionality without tests:
   • •Use git diff --name-only HEAD~5 to find recently changed files
   • •Cross-reference with coverage report
   • •Flag any new code in src/honeycomb/models/ or src/honeycomb/validation/ without 80%+ coverage

Coverage expectations by module:

• •models/ and validation/: Should be 80%+ (unit testable)
• •resources/: 30-50% typical (integration-heavy, tested via live tests)
• •cli/: Low coverage expected (tested manually)
• •tools/executor.py: 40-50% (tested via eval tests)

Step 6: Validate Documentation (requires direnv)

direnv exec . poetry run python scripts/validate_docs_examples.py
direnv exec . poetry run pytest tests/integration/test_doc_examples.py -v

Important: This step requires environment variables from .envrc. If direnv is not configured, skip with a warning.

After validation:

1. •Check for docs-code mismatch:
   • •Use git diff --name-only HEAD~5 to find changed source files
   • •Check if corresponding docs in docs/usage/ were updated
   • •Flag functionality changes without docs updates

Step 7: (Optional) Claude Tool Evaluation Tests

Only run if --eval argument is provided:

rm -rf tests/integration/.tool_call_cache/*.json
direnv exec . poetry run pytest tests/integration/test_claude_tools_eval.py -v -n 4

Warning: This step takes a long time (5-10 minutes) and requires ANTHROPIC_API_KEY.

Post-Pipeline Analysis

After all steps complete, provide a summary:

Coverage Analysis

1. •Overall coverage: Report the percentage
2. •Coverage gaps: List top 5 files with lowest coverage
3. •New code coverage: For files changed in recent commits, report their coverage

Documentation Sync Check

Compare changed files with docs:

# Get changed source files
git diff --name-only HEAD~5 -- 'src/honeycomb/*.py' 'src/honeycomb/**/*.py'

# Get changed doc files
git diff --name-only HEAD~5 -- 'docs/**/*.md'

Flag any source files in resources/, models/, or builders/ that changed without corresponding doc updates.

Test Coverage for New Features

For each new or significantly changed file:

1. •Check if it has test coverage
2. •If coverage < 70%, flag it
3. •Suggest what tests might be needed

Example Output Format

CI Pipeline Results
==================

Step 1: Install Dependencies ✓
Step 2: Format Code ✓ (2 files reformatted)
Step 3: Lint ✓ (3 issues auto-fixed)
Step 4: Type Check ✓
Step 5: Unit Tests ✓ (831 passed, 38% overall coverage)
Step 6: Documentation ✓
Step 7: Eval Tests ✓ (320 passed) [if --eval]

Coverage Analysis
-----------------
Overall: 38% (baseline for this project)

Models/Validation (target 80%+):
  ✓ src/honeycomb/models/tool_inputs.py: 99%
  ✓ src/honeycomb/models/tags_mixin.py: 100%
  ✓ src/honeycomb/validation/boards.py: 90%

Resources (30-50% expected):
  - src/honeycomb/resources/markers.py: 24%
  - src/honeycomb/resources/columns.py: 27%

Documentation Sync
------------------
Changed source files without doc updates:
  - src/honeycomb/resources/triggers.py (modified)
    Consider updating: docs/usage/triggers.md

New Code Without Tests
----------------------
  - src/honeycomb/models/new_feature.py: New file, 45% coverage
    Missing tests for: validate_feature(), get_feature()

Failure Handling

• •Lint failures (unfixable): Stop and report. User must fix manually.
• •Type errors: Stop and report. User must fix.
• •Test failures: Stop and report failed tests with tracebacks.
• •Doc validation failures: Report but continue if --skip-docs not set.

Quick Commands

# Full CI (default)
/ci

# CI with eval tests (for tool call changes)
/ci --eval

# Skip docs validation
/ci --skip-docs

# Just coverage analysis
/ci --coverage-only