Mutation Testing with mutmut
Mutation testing assesses test suite quality by introducing small changes (mutations) to source code and verifying that tests fail. If tests don't catch a mutation, it indicates gaps in test coverage or quality.
Key Concepts
- •Mutant: Modified version of code with small change
- •Killed: Test fails when mutation introduced (good)
- •Survived: Test passes despite mutation (test gap)
- •Mutation Score: Percentage of mutants killed
Quick Start
bash
# Install pip install mutmut # Run mutation testing mutmut run # View results mutmut results # Inspect specific mutant mutmut show 1 # Apply mutation to see the change mutmut apply 1 # Reset applied mutations mutmut apply 0
Configuration
ini
# setup.cfg or pyproject.toml [tool.mutmut] [mutmut] paths_to_mutate=src/ backup=False runner=python -m pytest -x tests_dir=tests/ dict_synonyms=Struct, NamedStruct
Example: Weak vs Strong Tests
python
# src/validators.py
def is_valid_order(order: dict) -> bool:
if not order:
return False
if not order.get("items"):
return False
if order.get("total", 0) <= 0:
return False
return True
def calculate_discount(total: float, tier: str) -> float:
if tier == "gold":
return total * 0.2
elif tier == "silver":
return total * 0.1
return 0.0
python
# ❌ Weak tests - mutations survive
def test_order_basic():
order = {"items": ["a"], "total": 10}
assert is_valid_order(order) == True # Only tests happy path
# ✅ Strong tests - kill mutations
def test_order_null():
assert is_valid_order(None) == False
assert is_valid_order({}) == False
def test_order_empty_items():
assert is_valid_order({"items": [], "total": 10}) == False
def test_order_zero_total():
assert is_valid_order({"items": ["a"], "total": 0}) == False
def test_order_negative_total():
assert is_valid_order({"items": ["a"], "total": -5}) == False
def test_order_valid():
assert is_valid_order({"items": ["a"], "total": 10}) == True
def test_discount_gold():
assert calculate_discount(100, "gold") == 20.0
def test_discount_silver():
assert calculate_discount(100, "silver") == 10.0
def test_discount_none():
assert calculate_discount(100, "bronze") == 0.0
Common Mutation Operators
| Original | Mutations |
|---|---|
> | >=, <, == |
< | <=, >, == |
== | != |
and | or |
+ | - |
return True | return False |
return x | return x + 1 |
Improving Mutation Score
When a mutant survives, add tests for:
- •Boundary conditions: Test at exact threshold values
- •Return values: Verify actual results, not just truthiness
- •All branches: Cover each conditional path
- •Edge cases: Empty, None, negative, zero
python
# Mutant survives: >= changed to >
def is_adult(age: int) -> bool:
return age >= 18
# ❌ Weak: age=25 won't catch >= to > mutation
def test_adult_weak():
assert is_adult(25) == True
# ✅ Strong: tests boundary
def test_adult_exactly_18():
assert is_adult(18) == True
def test_not_adult_17():
assert is_adult(17) == False
CI Integration
yaml
# .github/workflows/mutation.yml
name: Mutation Testing
on:
pull_request:
paths: ['src/**']
jobs:
mutation:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- run: pip install mutmut pytest
- run: mutmut run --CI
- run: mutmut results
Best Practices
- •Target critical business logic, not all code
- •Aim for 80%+ mutation score on important modules
- •Review survived mutants to improve tests
- •Test boundaries and edge cases thoroughly
- •Verify actual values, not just code execution