Cost-Optimized Log and Trace Sampling
This skill provides expertise in sampling strategies for managing observability costs.
Overview
Observability data can be expensive at scale. Smart sampling strategies maintain visibility while controlling costs.
Sampling Strategies
Head-Based Sampling
Decision made at trace start:
- •Probabilistic: Sample X% of traces randomly
- •Rate-limiting: Sample up to N traces per second
yaml
# OpenTelemetry config sampler: type: traceidratio ratio: 0.1 # Sample 10%
Tail-Based Sampling
Decision made after trace completes:
- •Error-based: Always sample traces with errors
- •Latency-based: Sample slow traces (> p99 latency)
- •Attribute-based: Sample specific user IDs or endpoints
Hybrid Approaches
- •Sample 100% of errors
- •Sample 100% of slow requests
- •Sample 10% of successful, fast requests
Log Sampling Strategies
- •Dynamic log levels: Debug in dev, warn in prod
- •Sample verbose logs: Log 1% of debug statements
- •Aggregate before sending: Count similar events locally
Cost Optimization Tips
- •Set appropriate retention periods
- •Use log aggregation to reduce volume
- •Filter noise before export
- •Use tiered storage (hot/warm/cold)
[Content to be expanded based on plugin_spec_agentient-observability.md specifications]