Test Before Deploying

Name: armor-test
Rating: 92
Author: anomalyarmor

Preview what alerts would fire with proposed configurations. Avoid alert fatigue by testing thresholds before enabling.

Prerequisites

•AnomalyArmor API key configured (~/.armor/config.yaml or ARMOR_API_KEY env var)
•Python SDK installed (pip install anomalyarmor)

When to Use

•"Test this freshness threshold before I enable it"
•"What alerts would fire if I set the threshold to 4 hours?"
•"Dry-run the schema drift check"
•"Preview the impact of this rule"
•"Will this configuration cause too many alerts?"

Steps

Dry-Run Freshness Threshold

•Specify the table and proposed threshold
•Call client.freshness.dry_run() with proposed config
•Review predicted alerts over historical data
•Adjust threshold if too many/few alerts predicted
•When satisfied, create the actual schedule

Preview Alert Rules

•Specify the event types to filter
•Call client.alerts.preview() with event types
•See what historical events would have triggered alerts
•Evaluate alert frequency
•Adjust configuration as needed

Example Usage

Dry-Run Freshness Threshold

python

from anomalyarmor import Client

client = Client()

# Test what would happen with a 4-hour freshness threshold
# Uses historical data to predict alert frequency
result = client.freshness.dry_run(
    asset_id="asset-uuid",
    table_path="public.orders",
    expected_interval_hours=4,
    lookback_days=7
)

print(f"Configuration: Alert if stale for {result.threshold_hours} hours")
print(f"Historical period: {result.lookback_days} days")
print()
print(f"Total checks analyzed: {result.total_checks}")
print(f"Would alert count: {result.would_alert_count}")
print(f"Alert rate: {result.alert_rate_percent:.1f}%")
print()

if result.would_alert_now:
    print(f"Current status: Would alert NOW (age: {result.current_age_hours:.1f}h)")
else:
    print(f"Current status: OK (age: {result.current_age_hours:.1f}h)")

print(f"\nRecommendation: {result.recommendation}")

Dry-Run Schema Drift Settings

python

from anomalyarmor import Client

client = Client()

# Preview schema drift detection with proposed settings
result = client.schema.dry_run(
    asset_id="asset-uuid",
    lookback_days=30
)

print(f"Schema Drift Dry-Run Results:")
print(f"  Total changes detected: {result.total_changes}")
print(f"  Changes by type: {result.changes_summary}")
print()

if result.sample_changes:
    print("Sample changes:")
    for change in result.sample_changes[:5]:
        print(f"  - {change.change_type}: {change.table_name}")

print(f"\nRecommendation: {result.recommendation}")

Preview Alert Rule Matches

python

from anomalyarmor import Client

client = Client()

# Preview what alerts would match a rule configuration
result = client.alerts.preview(
    event_types=["freshness_stale", "schema_drift"],
    severities=["critical", "warning"],
    lookback_days=7
)

print(f"Lookback: {result.lookback_hours} hours")
print(f"Alerts would match: {result.alerts_would_match}")
print(f"By severity: {result.alerts_by_severity}")
print(f"By type: {result.alerts_by_type}")
print()

if result.sample_alerts:
    print("Sample alerts:")
    for alert in result.sample_alerts[:5]:
        print(f"  - {alert.triggered_at}: {alert.message}")

Dry-Run Metrics Threshold

python

from anomalyarmor import Client

client = Client()

# Preview what anomalies would be detected with a metric
result = client.metrics.dry_run(
    asset_id="asset-uuid",
    table_path="catalog.schema.orders",
    metric_type="null_percent",
    column_name="email",
    sensitivity=1.5,
    lookback_days=14,
)

print(f"Metric Dry-Run: {result.metric_type}")
print(f"Total snapshots analyzed: {result.total_snapshots}")
print(f"Would alert count: {result.would_alert_count}")
print(f"Alert rate: {result.alert_rate_percent:.1f}%")
print()
print(f"Recommendation: {result.recommendation}")

Interpreting Results

Metric	Good Range	Action if Outside
Alert rate	5-25%	Adjust threshold up/down
Alerts per day	0.1 - 1.0	Adjust threshold up/down
Zero alerts	-	Tighten threshold

Follow-up Actions

•If results look good: Use /armor:monitor to enable the configuration
•If too many alerts: Increase threshold or add conditions
•If no alerts: Decrease threshold to catch real issues
•Compare different configurations to find optimal settings